Indonesian J our nal of Electrical Engineering and Computer Science V ol. 42, No. 1, April 2026, pp. 205 ∼ 214 ISSN: 2502-4752, DOI: 10.11591/ijeecs.v42.i1.pp205-214 ❒ 205 Complexity awar e cascade ar chitectur e f or impr o ving user satisfaction in con v ersational AI Constantinus Satrio 1 , De vi Fitrianah 2 1 Computer Science Department, BINUS Graduate Program – Master of Computer Science, Bina Nusantara Uni v ersity , Jakarta, Indonesia 2 Computer Science Department, Bina Nusantara Uni v ersity , Jakarta, Indonesia Article Inf o Article history: Recei v ed Aug 21, 2025 Re vised Jan 4, 2026 Accepted Mar 4, 2026 K eyw ords: Chatbot architecture Comple xity a w are cascade Con v ersational AI Retrie v al-augmented generation Service quality T ask completion rate User satisf action ABSTRA CT Con v entional task-oriented chatbots frequently suf fer from task incompletions and lo w user satisf action when handling comple x queries. This research intro- duces the comple xity a w are cascade , an adapti v e archite cture that impro v es user service quality by dynamically matching query comple xity with the appropri- ate computational response. The syste m uses condence and rele v ance scores to intelligently route requests through a sequence of a natural language under - standing (NLU) m odel, a retrie v al-augment ed generation (RA G) pipeline, or a lar ge language m odel (LLM). The tiered architecture w as e v aluated via a ran- domized controlled trial (RCT) with 150 participants, measuring task success and user satisf action. The full cascade achie v ed a 90% jour ne y completion rate, representing a 92.3% impro v ement o v er baseline system and substantial g ains in SER VQ U AL-based service quality scores. The e xperiment w as conducted in a domain-specic kno wledge base (essential oils) with a con v enience sam- ple that does not represent the global population, and no real-time deplo yment or long-term cost analysis w as performed. Accordingly , the ndings should be interpreted as e vidence of ef fecti v eness in a limited setting rather than as directly scalable to all domains. Ev en with these limitations, this study pro vides arigorously tested blueprint for de v eloping more rob ust and user -centric con v ersational AI systems. This is an open access article under the CC BY -SA license . Corresponding A uthor: Constantinus Satrio Computer Science Department, BINUS Graduate Program – Master of Computer Science Bina Nusantara Uni v ersity Jakarta, Indonesia Email: constantinus.satrio@binus.ac.id 1. INTR ODUCTION Con v ersational AI, particularly chatbots, has become a critical component of modern customer ser - vice systems [1]-[3]. While ef fecti v e at handling routine queries, con v entional chatbots e xhibit a signicant weakness: their inability to gracefully manage con v ersations outside their pre-dened training scope [4], [5]. This brittleness leads to high rates of con v ersational f ai lure, user frustration, and ultimately , a ne g ati v e im- pact on customer satisf action [6]. Existing litera ture of multi-stage systems, such as the static, k e yw ord-based clarication mechanism from Lautraite et al. [7], often lack dynamic, automated strate gies. Frame w orks lik e PromptChainer [8], for inst ance, sho w the potential of chaining LLMs b ut require manual programming. This re v eals a research g ap for an intelligent architecture that automatically manages con v ersational comple xity to impro v e user outcomes [9], [10]. J ournal homepage: http://ijeecs.iaescor e .com Evaluation Warning : The document was created with Spire.PDF for Python.
206 ❒ ISSN: 2502-4752 Prior studies on customer -service chatbots ha v e focused primarily on ho w interaction quality and in- terf ace design dri v er user satisf action, ofte n treating the underlying system architectures as a ”black box”. Research by Adam et al. [1] and Hsu and Lin [11] demonstrates ho w percei v ed service quality , social pres- ence, and con v ersational quality inuence user compliance and lo yalty , while other w ork e xamines the roles of anthropomorphism and personality in user management [3], [5], [6], [10], [12]-[15]. Ho we v er , these studies typically implement con v ersational engines as single stage systems, lea ving architectural choices such as multi- stage routing, retrie v al augmentation, and LLM inte gration lar gely une v aluated as primary determinants of user e xperience. Systematic re vie ws further highlight a concentration on technical performance or heterogeneous satisf action scales, with limited at tention to ho w specic architectural congurations af fect percei v ed service quality . This highlights a lack of controlled e xperiments that jointly analyze task success, service quality per - ceptions, and system latenc y across alternati v e system designs [2], [4], [9], [16]. Architecturally , the proposed comple xity a w are cascade (CA C) b uilds on prior multi-stage con v er - sational systems and retrie v al-augmented language models. While e xisting frame w orks demonstrate that de- composing comple x queries into stages impro v es answer quality and tools lik e PromptChainer enable comple x applications through prompt-chaining [7], [8], these approaches often rely on x ed routing logic or manual con- guration. Similarly , models such as RA G and REALM sho w ho w signicant performance g ains in kno wledge- intensi v e tasks, yet the y are typically e v aluated using technical benchmarks rather than human-centered service quality [17], [18]. This study addresses these g aps by operationalizing a CA C that inte grates RASA, RA G, and LLM into a dynamic pipeline, linking objecti v e journe y completion rates (JCR) and response times to SER VQ U AL-based user satisf action. Our results indicate that routing medium and high comple xity queries through deeper architectural stages impro v es both JCR and percei v ed service quality compared to single-stage baselines. The LLM stage primarily enhances user e xperience rather than pro viding statistically signicant g ains in task completion. In this conte xt, user e xperience e xtends be yond task completion, as prior research demonstrates that percei v ed service quality encompassing reliabi lity , responsi v eness, assurance, and empath y is a vital predictor of satisf action, trust, and continued use. These ndings align with the broader service-quality literature where SER VQ U AL-inspired scales are used to quantify t echnology-mediated services and e xplain outcomes such as lo yalty and compliance [6]. Moti v ated by this body of w ork, the present study adopts a SER VQ U AL-based instrument to e v aluate the chatbot as a service interf ace, measuring success by ho w well it manages e xpecta- tions, uncertainty , and trust alongside task success. This paper contrib utes by proposing the Comple xity A w are Cascade architecture for adapti v e query routing based on comple xity and safety constraints, implementing an end-to-end prototype using RASA, a grounded RA G pipeline, and a safety-conscious LLM as a cost-a w are ”g atek eeper”, and conducting a randomized controlled trial with 150 participants to compare three architectures (NLU-only , NLU+RA G, and CA C with LLM). Finally , we analyze the trade-of fs between task completion, user satisf action, safety , and computational cost, demonstrating that while the LLM stage primarily enhances user e xperience rather than ra w task success, it pro vides a v alidated blueprint for b u i lding more ef fecti v e, trust-based con v ersational AI [12]-[19]. 2. PR OPOSED COMPLEXITY A W ARE CASCADE ARCHITECTURE The CA C architecture is designed to manage con v ersational w orko ws by matching query comple xity with the appropriate le v el of computational po wer [15]. The CA C consists of three sequential stages, orches- trated by a dynamic routing logic. 2.1. Ar chitectur e stages Stage 1: Specialized NLU (RASA). The rst stage acts as a f ast, computationally ine xpensi v e g ate- k eeper . It utilizes the RASA frame w ork to handle most in-scope, task-oriented user int ents and i s optimized for speed and ef cienc y on kno wn con v ersational paths [20]. Stage 2: Retrie v al-augmented generation (RA G). If the RASA model’ s NLU condence is belo w a threshold (e.g., < 0 . 75 ), the system triggers a RA G pipeline [17], [18]. The RA G process in v olv es se v eral k e y steps, starting with an initial semantic retrie v al from a v ector database, follo wed by TF-IDF rele v ance ltering to re-rank candidates based on k e yw ord rele v ance and ensure conte xt precision. A cross-encoder then performs a joint, deep analysis of query-document pairs to pro vide a deterministic rele v ance score that is combined with the initial NLU condence to determine escalation to the LLM or rejection of out-of-scope queries for safety . T o maintain operational ef c ienc y , principal component analysis (PCA) is utilized for conte xt compression to Indonesian J Elec Eng & Comp Sci, V ol. 42, No. 1, April 2026: 205–214 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 207 pre v ent windo w o v ero w in the nal generation step, and 4-bit quantization is emplo yed to signicantly reduce memory footprint and accelerate inference without sacricing performance. Stage 3: LLM-based response generation. The curated conte xt from the RA G pipeline is passed via the LangChain frame w ork to a lar ge-scale generati v e model (e.g., ChatGPT) [21], [22]. This nal stage synthesizes the information to generate a coherent, conte xtually grounded response. This tiered management ensures that the most computationally e xpensi v e resource—the generati v e LLM—is used only when necessary and is al w ays supplied with a high-quality , optimized conte xt [23]. 2.2. Implementation details The system, b uilt with Pyt ho n [24] and Flask [25], uses a block ed randomization scheme. P artici pants were randomly assi gned to one of the three conditions RASA ONL Y , RASA RA G, RASA RA G LLM. This ensure equal group sizes (n=50 per group). Stage switching is triggered when RASA ’ s condence is belo w 0.75 or on a f allback. The RA G pipeline uses nomic-ai/nomic-embed-te xt-v1.5 [26] for embeddings, cross- encoder/ms-marco-MiniLM-L-6-v2 [27] as cross encoder , and a quantized unsloth/Phi-3-mini-4k-instruct-bnb- 4bit for generation [28]. The nal LLM stage uses OpenAI’ s gpt-4o-mini, incorporating chat history for con- te xt. All interactions are logged in a PostgreSQL dat abase with the pgv ector e xtension for analysis. The system escalates from R ASA to RA G if condence is belo w 0.75 or on a f allback. The nal LLM prompt combines retrie v ed conte xt with chat history for a more coherent response. Hando v ers are go v erned by real-t ime quantitati v e metrics [29]. Escalation from Stage 1 to 2 is t rig- gered by a lo w NLU condence score. Decision to escalate from Stage 2 to 3 is determined by cross-encoder rele v ance score, which pro vides deterministic judgment for the retrie v ed documents’ rele v ance t o the user’ s query [30]. This second metric allo ws the system to intelligently decide if the retrie v ed information is suf - cient or if the full po wer of the generati v e model is required. The system architecture (Figure 1) rst routes all queries to Stage 1 for RASA NLU assessment. A condence score acts as the initial g ate: high-condence queries are answered directly , while lo w-condence ones escalate to the Stage 2 RA G pipeline for conte xt retrie v al and re-ranking. A second g ate then uses the cross-encoder’ s rele v ance score to judge the retrie v ed conte xt. If the conte xt is highly rele v ant, the local quantized LLM responds. Only when both NLU condence and conte xt rele v ance are lo w does the query escalate to Stage 3, where the gpt-4o-mini model handles the most comple x queries using the curated conte xt and con v ersation history . 3. METHOD T o v alidate the proposed architecture, we emplo yed a randomized controll ed t rial (RCT) with a focus on user -centric outcomes [31]. 3.1. Experimental design T o systematically e v aluate each component’ s contrib ution, the e xperiment w as designed as an ab- lation study . P articipants were randomly assigned to one of three system congurations: (1) Condition A (single-stage): A weak baseline chatbot using only the RASA frame w ork; (2) Condition B (tw o-stage): An in- termediate system where RASA f alls back to the RA G pipeline for unresolv ed queries, then presenting retrie v ed conte xt directly to the user; (3) Condition C (multi-stage): The complete three stage architect ure, including the nal generati v e LLM. This design allo ws for a di rect comparison of the performance g ains from the RA G retrie v al stage (A vs. B) and the subsequent generati v e synthesis stage (B vs. C) [32]. 3.2. P erf ormance metrics W e e v aluated the system’ s impact on user e xperience with tw o primary metrics: (1) Journe y Com- pletion Rate (JCR): A measure of task succes s, where cate gorical ratings were con v erted to numerical scores “Y a, semua tug as selesai deng an baik. ” = 100 (complete success), “Han ya beberapa tug as yang selesai” = 40 (partially successful), T idak, saya meng alami ban yak masalah = 20 (mostly unsuccessful) [16]; (2) User Satis- f action (SER VQ U AL): Measured using the SER VQ U AL frame w ork. The surv e y comprised 16 items rated on a 5-point Lik ert scale (1 = Strongly Disagree to 5 = Strongl y Agree), grouped into v e dimensions: T angibles (3 items), Reliability (4 items), Responsi v eness (3 items), Assurance (3 items), and Empath y (3 items). A com- posite score for each dimension w as calcula ted for each participant by a v eraging the scores of its corresponding items [11]. Comple xity awar e cascade ar c hitectur e for impr o ving user satisfaction in ... (Constantinus Satrio) Evaluation Warning : The document was created with Spire.PDF for Python.
208 ❒ ISSN: 2502-4752 Figure 1. System o wchart of comple xity a w are cascade architecture 3.3. P articipant r ecruitment P articipants (N=150) were recruited via adv ertisements posted on a v ariety of online platforms, in- cluding uni v ersity student forums, general social media groups (e.g., F acebook, Instagram), and community messaging boards (e.g., WhatsApp) to g ather a di v erse sample, encompassing a wide spectrum of professional backgrounds and le v els of technical f amiliarity . Gender distrib ution is 60.97% male and 39.3% female. P ar - ticipants were randomly assigned to one of the three conditions (n=50 per group). All chat logs and surv e y responses were systematically recorded. 3.4. P articipant pr ocedur e and scenarios A standardized procedure w as implemented through a rdedicated web interf ace. P articipants were rst briefed and then guided through v e distinct task scenarios of v arying comple xity: (1) Simple query: A straightforw ard f act-based question; (2) Medium comple xity Requiring multi-step reasoning wit hin a kno wn domain; (3) Comple x query: A question requiring synthesis of information from multiple topics; (4) Out-of- conte xt query: A question designed to be be yond the base chatbot’ s training data; (5) Creati v e query: A task requiring the generation of original te xt or ideas. F or each scenario, participants were pro vided with e xample prompts and a ”cop y to clipboard” func- tion to standardize input phrasing across conditions. After interacting with the chatbot for all scenarios, partic- ipants completed the JCR and SER VQ U AL questionnaire. 3.5. Data analysis Due to the ordinal nature of Lik ert scale data and the signicant ne g ati v e sk e w and non-normal distrib ution identied in the SER VQ U AL responses (particularly in Condition A), non-parametric statistical tests were emplo yed. Group comparisons: The Kruskal-W allis H test, a non-parametric alternati v e to one- w ay ANO V A, w as used to determine if there were statistically signicant dif ferences in the median JCR and Indonesian J Elec Eng & Comp Sci, V ol. 42, No. 1, April 2026: 205–214 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 209 SER VQ U AL dimension scores acros s the three conditions. If a signicant dif ference w as found, Dunn’ s post- hoc test with Bonferroni correction w as applied for pairwise comparisons. Correlation analysis: The Spear - man’ s rank-order correlation coef cient ( ρ ) w as used to assess the monotonic relationship between the ordinal condition v ariable (A=1, B=2, C=3) and the user outcome metrics (JCR, SER VQ U AL scores). All analyses were conducted using Python (v3.12.3) with appropriate libraries (details can be seen on GitHub), and a p-v alue of < 0 . 05 w as considered statistically signicant. 4. RESUL TS AND DISCUSSION Analysis of the e xperimental data indicates a statistically signicant, positi v e correlation between the implementation of the comple xity a w are cascade architecture and the measured outcomes of task success and user satisf action. 4.1. T ask success (JCR) The distrib ution of JCR scores acros s the three architectures is summarized in T able 1. The tabl e reports mean JCR, as well as the proportion of participants who e xperienced complete v ersus f ailed journe ys under each conguration, allo wing a direct comparison of ho w architectural comple xity impacts task success. The analysis of mean JCR re v ealed a signicant impro v ement in task success as architectural comple xity in- creased. The mean JCR score increased from 46 .8% for the Single-Stage system to 77.2% for the tw o-stage system, and 90% mean JCR for the multi-stage architecture. T able 1. Mean journe y completion rate by chatbot architecture Architecture Mean JCR (%) Success rate (100%) F ailure rate ( < 50%) Single-stage 46.8 24.0 76.0 T w o-stage 77.2 68.0 32.0 Multi-stage 90 86.0 14.0 4.2. User satisfaction (SER VQ U AL analysis) The impro v ements in task succe ss translated directly to a higher perception of service quality as sho wn in T able 2 and Figure 2 . The median satisf action scores impro v ed progressi v ely from a baseline of 1.133 for the single-stage system to 2.417 for the multi-stage architecture. This substantial uplift demonstrates that system- le v el choices serv e as a direct le v er for enhancing user trust and perceptions of reliability . Furthermore, the multi-stage conguration achie v ed a more consistent user e xperience, demonstrated by a tighter interquartile range (IQR = 0.616) compared to the tw o-stage system (IQR = 0.842), suggesting that the nal generati v e stage pro vides a more stable and predictable service interf ace. T able 2. SER VQ U AL scores by chatbot architecture Architecture N Median IQR Single-stage 50 1.133 0.250 T w o-stage 50 1.842 0.842 Multi-stage 50 2.417 0.616 4.3. Statistical ndings T o formally test whether t hese observ ed dif ferences were statistically signicant, we applied non- parametric analyses suited to the ordinal and sk e wed nature of the data. T able 3 summarizes the main statistical ndings, including the Kruskal–W allis tests for o v erall architectural ef fects, the Spearman rank correlations between architectural comple xity and the outcome metrics, and the post-hoc comparison between the RA G- only and RA G+LLM conditions. Kruskal–W allis tests conrmed that architectural comple xity had a signicant ef fect on both user s at- isf action (SER VQ U AL, H=85.39, p < 0.001) and task completion (JCR, H=37.70, p < 0.001). Spearman’ s ρ further sho wed strong correlation between comple xity and satisf action ( ρ =0.748) and a moderate correlation with task s uccess ( ρ =0.490). Post-hoc analysis indicated that adding the LLM stage did not signicantly im- pro v e JCR o v er the RA G system alone (p=0.107), suggesting that its main contrib ution lies in enhancing the user e xperience rather than task success. Finally , the weak correlation between SER VQ U AL and JCR ( ρ =0.297) underscores that satisf action and success, while related, are distinct outcomes. Comple xity awar e cascade ar c hitectur e for impr o ving user satisfaction in ... (Constantinus Satrio) Evaluation Warning : The document was created with Spire.PDF for Python.
210 ❒ ISSN: 2502-4752 Figure 2. Box plots sho wing the distrib ution of o v erall SER VQ U AL scores for the single-stage, tw o-stage, and multi-stage architectures T able 3. K e y statistical ndings along with statistical test type K e y Finding Statistical T est Result p-v alue Ov erall architectural impact Kruskal-W allis H SER VQ U AL:H=85.39, JCR:H=37.70 < 0.001 Comple xity dri v es satisf action Spearman’ s ρ ρ = 0.748 (V ery Strong) < 0.001 Comple xity moderately impro v es task success Spearman’ s ρ ρ = 0.490 (Moderate) < 0.001 LLM enhances e xperience, not just success Post-hoc Dunn’ s test JCR Comparison: p = 0.107 0.1070 (RA G vs RA G+LLM) T ask Success ̸ = User Satisf action Spearman’ s ρ ρ = 0.297 (W eak) < 0.001 (SER VQ U AL ↔ JCR) 4.4. Discussion The study’ s ndings re v eal a substantial impro v ement in task success correlated with archit ectural comple xity . From a quantitati v e perspecti v e, the g ains achie v ed by the CA C are lar ge relati v e to those reported in prior chatbot research. Adam et al. [1] e xperimentally e xamined ho w interaction-design manipulations (foot- in-the-door requests and anthropomorphic design cues) af fect user compliance with a customer -service chatbot. In their study , compliance rose from 63% in the control condition to 77% with only foot -in-the-door cues, 84% with only anthropomorphic cues, and 95% when both were combined, an absolute g ain of 32 percentage points and a relati v e impro v ement of around 51% o v er the baseline [1]. In our randomized controlled trial, JCR increased from 46.8% in the single-stage baseline to 77.2% i n the tw o-stage architecture and 90.0% in the full CA C. This corresponds to an absolute g ain of 43.2 percentage points and a relati v e impro v ement of 92.3% in task completion. Whereas Adam et al. [1] attrib ute their ef fects to surf ace-le v el interaction design for a single request, our results sho w that system-le v el architectural changes can yield impro v ements of comparable or greater magnitude in end-to-end journe y completion across a multi-turn, information-seeking domain. A similar pattern emer ges when comparing our ndings with studies that focus on customer s atis- f action and lo yalty rather than task completion. Hsu and Lin [11] modelled satisf action and lo yalty to w ard customer -service chatbots using an e xtended e-service quality frame w ork, sho wing that AI chatbot service re- co v ery quality and con v ersational quality signicantly predict satisf action, and that core service quality and sat- isf action predict lo yalty . Their structural model reports strong paths from chatbot quality to satisf action and lo y- alty (standardized coef cients e xceeding 0.5) and e xplains a lar ge proportion of v ariance in lo yalty (R 2 abo v e Indonesian J Elec Eng & Comp Sci, V ol. 42, No. 1, April 2026: 205–214 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 211 0.60), b ut it does not include an e xplicit task-completi on or journe y-complet ion metric. In c on t rast, our e xper - iment links architect ural conguration, objecti v e JCR, and SER VQ U AL scores within a single RCT : the full CA C conguration simultaneously impro v es task completion by 92.3% relati v e to the baseline and produces a substantial uplift in SER VQ U AL-based percei v ed service quality . In other w ords, our results complement Hsu and Li n ’ s construct-le v el ndings by sho wing that a concrete architectural design choice—comple xity-a w are cascading—can be treated as a le v er for jointly impro ving service quality and completion outcomes, not only for shifting latent satisf action and lo yalty scores. Prior e xperimental w ork on chatbot user e xperience also tends to report ef fects primarily at the per - ceptual le v el. Haugeland et al. [3], for e xample, conducted a randomized e xperiment (n = 35) comparing topic-led v ersus task-led con v ersations and b utton-based v ersus free-te xt interaction in a customer -service chat- bot. The y found that topic-led con v ersations signicantly increased percei v ed anthropomorphism and hedonic quality , while b utton-based interaction impro v ed pragmatic and hedonic quality , all measured on se v en-point UX scales. Ho we v er , their study does not report changes in objecti v e task success or completion rates, and the sample size is relati v ely small compared to our RCT with 150 participants. Our ndings e xtend this line of w ork by sho wing that architectural comple xity—operati o na lized as a dual-threshold cascade o v er RASA, RA G, and LLM s tages—not only af fects user perceptions (SER VQ U AL) b ut also leads to lar ge, quantiable g ains in end-to-end journe y completion. Systematic re vie ws of user -e xperience assessment with con v ersational agents remain relati v ely small in scope and highlight a fragmented e v aluation la n ds cape. F or e xample, T ubin et al. [16] systematically searched four major HCI-related databases (A CM, IEEE, Springer , and Scopus) and initially identied 482 papers, b ut only 27 studies met their inclusion criteria for reporting ho w user e xperience with con v ersational agents w as assessed. Their analysis sho ws a wide dispersion of methods, hea vy reliance on self-created post- test ques tionnaires, and v ery limited use of v alidated UX instruments or combined pre-, during-, a nd post-use assessments [16]. In parallel, Gamboa-Cruzado et al. [2] re vie w customer -service chatbots and document a fragmented landscape of application domains, technologies, and e v aluation practices, while more recent re vie ws of customer -service chatbot e xperience similarly synthesize dozens of empirical studies without iden- tifying man y that jointly analyze service-quality perceptions and hard performance metrics such as completion or resolution rates [9], [16]. Ag ainst this backdrop, the present study adds a relati v ely rare data point: a con- trolled, four -arm RCT that e xplicitly v aries the architecture (single stage vs. na ¨ ıv e stack vs. three- stage CA C vs. four -stage CA C with LLM), uses a structured SER VQ U AL-based instrument, and reports both JCR and SER VQ U AL outcomes. The size of the observ ed dif ferences—o v er 40 percentage points in JCR between base- line and full CA C—indicates that architectural choices can ha v e an impact on user outcomes comparable to or lar ger than those reported for interaction-design manipulations and service-quality perceptions alone. During the e xperiment, users occasionally entered of f-domain queries to the chatbot’ s designed kno wl- edge base of essential oils (e.g., geopolitical topics unrelated to essential oils). In these instances, the cross- encoder’ s rele v ance scoring acted as a safety system. By returning a near -zero condence score (e.g., 0.000047), it correctly identied the query as out-of-scope. Consequently , the chatbot refused to generate a speculati v e or potentially incorrect answer , instead i nforming the user of its limitations. While this is the desired beha viour for a trustw orth y AI, each refusal w as logged as an incomplete journe y , thereby p r e v ent ing the JCR from reaching a perfect score. This intentional refusal to answer irrele v ant queries e xplains the ceiling on the JCR and the SER VQ U AL scores. It represents a deliberate trade-of f: sacricing a perfect task completion metric to ens ure high Reliabil- ity and Assurance. Users percei v e the system as more trustw orth y not only because it pro vides correct answers b ut also because it is honest about what it does not kno w . This conrms that task success and user satisf action are not strictly correlated, underscoring the importance of reliability and transparenc y as independent dri v ers of user e xperience. This underscores a k e y design principle: optimizing for user -f acing outcomes of safety and reliability is more v aluable than stri ving for a a wless b ut potentially misleading completion rate. 4.5. Limitations This study has se v eral important limitations that must be considered when interpreting the res ults. First, the e xperiment w as conducted in a single, domain-specic kno wledge base (ess ential oils) using scripted scenarios. As a result, the ndings may not be generalize to high-stak es domains lik e nance or healthcare. Second, participants recruitment via uni v ersity forums and social media resulted in a con v enience sample that is relati v ely homogeneous and does not represent the global population. Most critically , the research lack ed long- Comple xity awar e cascade ar c hitectur e for impr o ving user satisfaction in ... (Constantinus Satrio) Evaluation Warning : The document was created with Spire.PDF for Python.
212 ❒ ISSN: 2502-4752 term e xploration, as no real-time production deplo yment trials or lar ge-scale computati onal cost studies were performed to measure long-term user retention or infras tructure costs. While the methodological contrib ution lies in the architectural design rather than a ne w model class, the ndings pro vide a v alidated blueprint for practical application. Future w ork should e v aluate the architecture across multiple domains with more di v erse and representati v e samples under eld conditions that capture long-term beha vior and operational constraints. Furthermore, it is recommended that practitioners adopt the CA C as an incremental layer , utilizing the dual- threshold routing to reserv e e xpensi v e LLM calls for comple x cases while letting the NLU or RA G tiers handle routine queries to balance satisf action and cost in lar ge-scale customer service en vironments. 5. CONCLUSION This research introduced and empirically v alidated the CA C, a system architecture for managi n g con- v ersational AI w orko ws. Through a RCT with 150 participants, we demonstrated that this multi-stage f all- back system signicantly outperforms simpler architectures in its ability to complete user tasks and impro v e user -percei v ed service quality . Ultimately , this research pro vides a v alidated blueprint for b uilding more ef- fecti v e con v ersational AI, demonstrating that true user -centric design in v olv es a careful balance between task completion and the system’ s ability to earn user trust by safely managing its o wn limitations. The principles demonstrated here can inform future research into de v eloping safety-grounded, comple xity-a w are con v ersa- tional AI for practical deplo yment. F or practitioners, these ndings suggest that a comple xity-a w are cascade can be adopted as an incremental layer on top of e xisting task-oriented chatbots. In lar ge-scale customer -service en vironments, the dual-threshold routing logic can be used to reserv e LLM calls for genuinely comple x or am- biguous cases, while letting a f ast NLU or RA G tier handle routine queries, thereby balancing user satisf action and infrastructure cost. The architecture can also be inte grated with e xisti ng tick eting systems and kno wledge bases by treating each stage as a modular service that e xposes standardized interf aces for logging, escalation, and monitoring. Future deplo yments shoul d instrument both task-le v el metrics (e.g., resolution rates, hand-of f to human agents) and service-quality indicators (e.g., SER VQ U AL dimensions) to continuously tune thresholds and escalation policies. Extending this e v al uation to multiple domains and long-running production systems is a natural ne xt step to v alidate ho w well t he CA C scales be yond the controlled e xperimental setting reported here. A CKNO WLEDGMENTS The authors ackno wledge the use of AI tools to assi st in the preparation of this manuscript. As sistance w as pro vided for code generation and deb ugging (Anthropic’ s Claude) and for editorial support, including language polishing and ensuring conciseness (Google’ s Gemini). The authors retain full responsibility for the nal content, all interpretations, and the inte grity of the w ork. FUNDING INFORMA TION This w ork w as supported and funded by the Binus Graduate Program, Bina Nusantara Uni v ersity . A UTHOR CONTRIB UTIONS ST A TEMENT This journal uses the C ontrib utor Roles T axonomy (CRediT) to recognize indi vidual author contrib u- tions, reduce authorship disputes, and f acilitate collaboration. Name of A uthor C M So V a F o I R D O E V i Su P Fu Constantinus Satrio ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ De vi Fitrianah ✓ ✓ ✓ ✓ ✓ ✓ ✓ C : C onceptualization I : I n v estig ation V i : V i sualization M : M ethodology R : R esources Su : Su pervision So : So ftw are D : D ata Curation P : P roject Administration V a : V a lidation O : Writing - O riginal Draft Fu : Fu nding Acquisition F o : F o rmal Analysis E : Writing - Re vie w & E diting Indonesian J Elec Eng & Comp Sci, V ol. 42, No. 1, April 2026: 205–214 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 213 CONFLICT OF INTEREST ST A TEMENT Authors state no conict of interest. D A T A A V AILABILITY The anon ymized summary dataset and the computational notebook that support the ndings of this study are openly a v ailable in https://doi.or g/10.5281/zenodo.18839338. Additional data, including the detailed anon ymized chat logs and participant-le v el assignments, are not publicly a v ailable to protect participant pri v ac y b ut can be obtained from the corresponding author upon reasonable request. REFERENCES [1] M. Adam, M. W essel, and A. Benlian, “ Ai-based chatbots in customer service and their ef fects on user compliance, ” Electr onic Mark ets , v ol. 31, no. 2, pp. 427–445, Jun. 2021. [2] J. Gamboa-Cruzado et al. , “Chatbots for customer service: A comprehensi v e systematic literature re vie w , ” J ournal of Theor etical and Applied Information T ec hnolo gy , v ol. 15, p. 19, 2022, accessed: Aug. 19, 2025. [Online]. A v ailable: http://www .jatit.or g/v olumes/V ol100No19/16V ol100No19.pdf [3] I. K. F . Haugeland, A. F ølstad, C. T aylor , and C. Ale xander , “Understanding the user e xperience of customer service chatbots: An e xperimental study of chatbot interaction design, ” International J ournal of Human-Computer Studies , v ol. 161, p. 102788, May 2022. [4] M. A. K uhail, N. Alturki, S. Alramla wi, and K. Alhejori, “Interacting with educational chatbots: A systematic re vie w , ” Education and Information T ec hnolo gies , v ol. 28, no. 1, pp. 973–1018, Jan. 2023. [5] X. Xing, M. Song, Y . Duan, and J. Mou, “Ef fects of dif ferent service f ailure types and reco v ery strate gies on the consumer response mechanism of chatbots, ” T ec hnolo gy in Society , v ol. 70, p. 102049, Aug. 2022. [6] E. Svikhnushina, A. Placinta, and P . Pu, “User e xpectations of con v ersational chatbots based on online re vie ws, ” in Pr oceedings of the 2021 A CM Designing Inter active Systems Confer ence , Jun. 2021, pp. 1481–1491. [7] H. Lautraite, N. Naji, L. Marceau, M. Queudot, and E. Charton, “Multi-stage clarication in con v ersational ai: The case of question- answering dialogue systems, ” Oct. 2021, accessed: Aug. 19, 2025. [Online]. A v ailable: https://arxi v .or g/pdf/2110.15235 [8] T . W u et al. , “Promptchainer: Chaining lar ge language model prompts through visual programm ing, ” in Pr oceedings of the CHI Confer ence on Human F actor s in Computing Systems , Apr . 2022. [9] J. Deriu et al. , “Surv e y on e v aluation methods for dialogue systems, ” Articial Intellig ence Re vie w , v ol . 54, no. 1, pp. 755–810, Jan. 2021. [10] G. R. S. Silv a and E. D. Canedo, “T o w ards user -centric guidelines for chatbot con v ersational design, ” International J ournal of Human–Computer Inter action , v ol. 40, no. 2, pp. 98–120, 2024. [11] C . L. Hsu and J. C. C. Lin, “Understanding the user satisf action and lo yalty of customer service chatbots, ” J ournal of Retailing and Consumer Services , v ol. 71, p. 103211, Mar . 2023. [12] L. M. de Cosmo, L. Piper , and A. Di V ittorio, “The role of attitude to w ard chatbots and pri v ac y concern on the relationship between attitude to w ard mobile adv ertising and beha vioral int ent to use chatbots, ” Italian J ournal of Mark eting , v ol. 2021, no. 1–2, pp. 83–102, Jun. 2021. [13] J. Moilanen, A. V isuri, S. A. Suryanarayana, A. Alorwu, K. Y atani, and S. Hosio, “Measuring the ef fect of mental health chatbot personality on user eng agement, ” in A CM International Confer ence Pr oceeding Series , No v . 2022, pp. 138–150. [14] V . T a et al. , “User e xperiences of social support from companion chatbots in e v eryday conte xts: Thematic analysis, ” J ournal of Medical Internet Resear c h , v ol. 22, no. 3, p. e16235, Mar . 2020. [15] J. Chen, F . Guo, Z. Ren, M. Li, and J. Ham, “Ef fects of anthropomorphic design cues of chatbots on users’ perception and visual beha viors, ” International J ournal of Human–Computer Inter action , v ol. 40, no. 14, pp. 3636–3654, 2024. [16] C. T ubin, J. P . Mazuco Rodriguez, and A. C. B. de Marchi, “User e xperience with con v ersational agent: a systematic re vie w of assessment methods, ” Behaviour & Information T ec hnolo gy , v ol. 41, no. 16, pp. 3519–3529, Dec. 2022. [17] P . Le wis et al. , “Retrie v al-augmented generation for kno wledge-intensi v e nlp tasks, ” in Advances in Neur al In- formation Pr ocessing Systems , v ol. 33, 2020, pp. 9459–9474, accessed: Aug. 19, 2025. [Online]. A v ailable: https://github .com/huggingf ace/transformers/blob/master/ [18] K. Guu, K. Lee, Z. T ung, P . P asupat, and M.-W . Chang, “Retrie v al augmented language model pre-training, ” in Pr o- ceedings of the 37th International Confer enc e on Mac hine Learning , 2020, accessed: Aug. 19, 2025. [Online]. A v ailable: https://proceedings.mlr .press/v119/guu20a.html [19] C. Prentice and M. Nguyen, “Eng aging and retaining customers with ai and emplo yee service, ” J ournal of Retailing and Consumer Services , v ol. 56, p. 102186, Sep. 2020. [20] %BIBentryAL T interw ordspacing R. T echnologies, “Rasa: Open source language understanding and dialogue management, ” Dec. 2017, accessed: Aug. 19, 2025. [Online]. A v ailable: https://arxi v .or g/pdf/1712.05181 [21] “langchain-ai/langchain: Build conte xt-a w are reasoning applications, ” accessed: Aug. 19, 2025. [Online]. A v ailable: https://github .com/langchain-ai/langchain [22] “Chatgpt, ” accessed: Aug. 21, 2025. [Online]. A v ailable: https://chatgpt.com/ [23] T . B. Bro wn et al. , “Language models are fe w-shot learners, ” in Advances in Neur al Information Pr ocessing Systems , v ol. 33, 2020, pp. 1877–1901, accessed: Aug. 19, 2025. [Online]. A v ailable: https://commoncra wl.or g/the-data/ [24] “W elcome to p ython.or g, ” accessed: Aug. 21, 2025. [Online]. A v ailable: https://www .p ython.or g/ [25] “pallets/ask: The p ython micro frame w ork for b uilding web applications, ” accessed: Aug. 21, 2025. [Online]. A v ailable: https://github .com/pallets/ask Comple xity awar e cascade ar c hitectur e for impr o ving user satisfaction in ... (Constantinus Satrio) Evaluation Warning : The document was created with Spire.PDF for Python.
214 ❒ ISSN: 2502-4752 [26] “nom ic-ai/nomic-embed-te xt-v1.5, ” accessed: Aug. 21, 2025. [Online]. A v ailable: https://huggingf ace.co/nomic-ai/nomic-embed- te xt-v1.5 [27] “c ross-encoder/ms-marco-minilm-l6-v2, ” accessed: Aug. 21, 2025. [Online]. A v ailable: https://huggingf ace.co/cross-encoder/ms- marco-MiniLM-L6-v2 [28] “uns loth/phi-3-mini-4k-instruct-bnb-4bit, ” accessed: Aug. 21, 2025. [Online]. A v ailable: https://huggingf ace.co/unsloth/Phi-3- mini-4k-instruct-bnb-4bit [29] S . Roller et al. , “Recipes for b uilding an open-domain chatbot, ” in Pr oceedings of the 16th Confer ence of the Eur opean Chapter of the Association for Computational Linguistics , Apr . 2021, pp. 300–325. [30] S. Humeau, K. Shuster , M.-A. Lachaux, and J. W eston, “Poly-encoders: T ransformer architectures and pre-training strate gies for f ast and accurate multi-sentence scoring, ” Apr . 2019, accessed: Aug. 19, 2025. [Online]. A v ailable: https://arxi v .or g/pdf/1905.01969 [31] C. Schillings, D. Meissner , B. Erb, D. Schultchen, E. Bendig, and O. Pollatos, “ A chatbot-based interv ention with elme to impro v e stress and health-related parameters in a stressed sample, ” F r ontier s in Digital Health , v ol. 5, p. 1046202, Mar . 2023. [32] I. Hameed et al. , “Based-xai: Breaking ablation studies do wn for e xplainable articial intelligence, ” Jul. 2022, accessed: Aug. 19, 2025. [Online]. A v ailable: https://arxi v .or g/pdf/2207.05566 BIOGRAPHIES OF A UTHORS Constantinus Satrio is a graduate student in the Master of Computer Science program at Bina Nusantara Uni v ersity . He holds bachelor’ s de grees in computer applications and computer science. Prior to his postgraduate studies, he established a rob ust career in the tech industry , holding roles as a softw are engineer , senior data engineer , and database administrator . His current research focuses on the practical application of Articial Intelligence, with a particular specialization in b uild- ing ef cient and scalable Retrie v al Augmented Generation (RA G) systems. His w ork aims to bridge the g ap between lar ge scale industrial systems and cutting-edge AI research, le v eraging his industry background to inform academic inquiry . As the lead author for thi s study , he w as the primary architect of the Comple xity A w are Cascade system, responsible for its conceptual design, end-to-end softw are implementation, and the e x ecution a nd analysis of the e xperimental results. He can be contacted at email: constantinus.satrio@binus.ac.id. De vi Fitrianah is an Associate Professor at the Master of Computer Science Department at Bina Nusantara Uni v ersity . She recei v ed her bachelor’ s de gree in computer science from Bina Nusantara Uni v ersity , follo wed by a master’ s de gree in information technology and a Ph.D. in Com- puter Science from Uni v ersitas Indonesi a in 2008 and 2015, respecti v ely . In 2014, she w as a w arded a sandwich program placement at Michig an State Uni v ersity , USA. Her research interests include Data Mining, Machine Learning, Articial Intelligence, and Applied Remote Sensing. F or this study , Dr . De vi serv ed as the principal in v estig ator and research supervisor . She guided the conceptualization of the cascade architecture, pro vided critical insights into the methodology , and o v ersa w the nal analysis and v alidation of the ndings. She can be contacted at email: de vi.trianah@binus.ac.id. Indonesian J Elec Eng & Comp Sci, V ol. 42, No. 1, April 2026: 205–214 Evaluation Warning : The document was created with Spire.PDF for Python.