IAES Inter national J our nal of Articial Intelligence (IJ-AI) V ol. 14, No. 4, August 2025, pp. 3375 ∼ 3385 ISSN: 2252-8938, DOI: 10.11591/ijai.v14.i4.pp3375-3385 ❒ 3375 A r einf or cement lear ning paradigm f or V ietnamese aspect-based sentiment analysis V iet The Bui 1 , Linh Thuy Ngo 2,3 , Oanh Thi T ran 2 1 School of Computing and Information Systems, Sing apore Management Uni v ersity , Sing apore, Sing apore 2 International School, V ietnam National Uni v ersity Hanoi, Hanoi, V ietnam 3 Banking Academy of V ietnam, Hanoi, V ietnam Article Inf o Article history: Recei v ed Sep 3, 2024 Re vised Feb 19, 2025 Accepted Mar 15, 2025 K eyw ords: Aspect-based sentiment analysis Deep learning Pre-trained models Reinforcement learning V ietnamese ABSTRA CT This paper present s the task of aspect-based sentiment analysis (ABSA) that rec- ognizes the sentiment polarity associated with each aspect of entities discussed in custom ers’ re vie ws, focusing on a lo w-resourced language, V ietnamese. Un- lik e con v entional classication approaches, we le v erage reinforcement learning (RL) techniques by formulating the task as a Mark o v decision process. This ap- proach allo ws an RL agent to handle the hierarchical nature of ABSA, sequen- tially predicting entities, aspects, and sentiments by e xploiting re vie w features and pre viously predicted labels. The agent seeks to disco v er optimal policies by maximizing cumulati v e long-term re w ards through accurate entity , aspect, and sentiment predictions. The e xperi mental results on public V ietnamese datasets sho wed that the proposed approach yielded ne w state of the art (SO T A) results in both hotel and restaurant domains. Using the best model, we achie v ed an impro v ement of 1% to 3% in the F 1 scores for detecting aspects and the corre- sponding sentiment polarity . This is an open access article under the CC BY -SA license . Corresponding A uthor: Oanh Thi T ran International School, V ietnam National Uni v ersity Hanoi 144 XuanThuy , CauGiay , Hanoi, V ietnam Email: tranthioanh@vnu.edu.vn 1. INTR ODUCTION Aspect-based sentiment analysis (ABSA) in v olv es identifying sentiments directed to w ards speci c aspects of enti ties mentioned in te xts. It has been widely applied in monitoring brand reputation, analyzing customer feedback, and tracking public sentiment to w ards social and political e v ents. F ormally , gi v en a te xt, a prediction model is b uilt to determine the set of triples < e, a, p > , where e represents entities, a denotes the associated aspects, and p indicates the sentiment polarity . Figure 1 illustrates a customer re vi e w of four k e y triples which are R OOM, CLEANLINESS, positi v e; SER VICE, GENERAL, positi v e; FOOD&DRINK, Q U ALITY , positi v e; and FOOD&DRINK, STYLE&OPTION, ne g ati v e. The structure of ABSA prediction is inherently hierarchical, reecting the natural progression of sentiment e xpressi on: rst, entities are identied; ne xt, aspects related to each entity are e xtracted; and nally , the sentiment assoc iated with each aspect is determined. This order - entities → aspects → polarity enables each prediction stage to le v erage the conte xt established in the preceding stages, creating a more ac- curate and conte xt-sensiti v e sentiment analysis. Until no w , con v entional approaches ha v e framed the task as a classication problem, using independent or joint learning methods. Researchers ha v e e xplored incorporating deep learning methods, such as recurre nt neural netw orks [1], [2], con v olution neural net w orks [3], trans- J ournal homepage: http://ijai.iaescor e .com Evaluation Warning : The document was created with Spire.PDF for Python.
3376 ❒ ISSN: 2252-8938 former models [4]-[7], attention models [8], and multi-task solutions [9] to capture the conte xtual and semantic nuances of input te xts for sentiment analysis. These techniques ha v e sho wn prom ising results in capturing ne- grained sentiment information, unfortunately , the y do not well capture the conte xt-dependent dependencies among entities, aspects, and polarity . Figure 1. A sample re vie w’ s hierarchical structure, annotated with entities, their aspects, and polarity Recently , reinforcement learning (RL) [10] has sho wn promise in v arious domains such as robotics [11], wireless netw orks [12], and g ame [13]. Ho we v er , its application to natural language processing (NLP) tasks remains lar gely restricted t o con v ersational systems [14]. Gi v en RL 's unique strength in handling sequen- tial decision-making, it presents a promising approach for ABSA. By treating ABSA as a Mark o v decision pro- cess, ABSA ’ s hierarchical structure - progressing from entity recognition to aspect identication and, nally , to sentiment prediction - can be managed as a series of sequential decisions. In this setting, deep Q-learning is le v eraged to dynamically update state representations as it identies entities, assigns them to rele v ant as- pects, and nally predicts the sentiment for each aspect. Through optimizing these decisions for long-term re w ards, the approach is e xpected to impro v e classication accurac y by capturing the structured dependencies often missed in traditional classication methods. T o our kno wledge, no prior w ork has framed ABSA as an RL problem, especially for lo w-resource languages, making this a no v el direction in sentiment analysis. This paper mak es the follo wing contrib utions: i) introduces a ne w approach using RL to solv e the ABSA task with a specic focus on a lo w-resource language, a.k.a V ietnamese; and ii) e xtensi v ely conducts dif ferent e xperiments to v erify the ef fecti v eness of the proposed approach on tw o a v ailable benchmark datasets. The remainder of the paper is structured as follo ws. Section 2 describes related w ork. Section 3 sho ws ho w to treat ABSA as a Mark o v decision process. Then, section 4 mathematically designs an RL agent to solv e this Mark o v decision process. Section 5 presents the deep Q-learning to solv e the ABSA in the RL setting. Section 6 describes the public benchmark corpus, setups of e xperiments, e xperimental result s, and some discussion. Finally , we conclude the paper and point out some future lines of w ork in section 7. 2. RELA TED W ORK Research on sentiment analysis has been performed for dif ferent le v els such as document le v el, sen- tence le v el, phrase le v el, and aspect le v el [15]. There are tw o mainly used approaches to e xplore sentiments: le xicon-based approaches and machine learning approaches. The former one [16] is e xtremely feasible for sentiment analysis at the sentence and phrase le v els. The latter one enables systems to acquire ne w abilities without being e xplicitly programmed to do so. Some typical w ork is a nai v e Bayes model along with an sup- port v ector machine (SVM) model [17], semi-supervised machine learning that inte grates pre-processing and classication algorithms for unlabelled datasets [18], attention (CNN-RNN) [8]. In rece n t years, sentiment analysis has become more prominent, and ASB A is a v aluable and quickly e xpanding research eld. Comple x algorithms [3], [5], [7], [9], [19] lik e long short-t erm memory (LSTM), pre-trained models lik e BER T , and GPT -2 may be used to accomplish the task. These classication techniques ha v e sho wn promising results in Int J Artif Intell, V ol. 14, No. 4, August 2025: 3375–3385 Evaluation Warning : The document was created with Spire.PDF for Python.
Int J Artif Intell ISSN: 2252-8938 ❒ 3377 capturing ne-grained sentiment information. Unfortunately , the y ma y not ef fecti v ely capture the hierarchical and interdependent nature of aspect-based sentiments. RL [10] is a po werful machine learning technique where an agent learns to mak e decisions by inter - acting with an en vironment, recei ving re w ards or punishments, and adjusting its actions to maximize long-term cumulati v e re w ards. It is primarily used for sequential decision-making tasks rather than classication tasks. F or e xample, se v eral RL models ha v e been studied in planning and control problems [20], action-conditional video prediction [21], and wireless netw orks [12]. RL can be combined with other techniques, such as deep learning, to perform tasks in NLP such as sequence-to-sequence learning for te xt generation and classica- tion tasks [14], [22]. RL demonstrates promise in NLP classication tasks, especially those with multiple or hierarchical labels, where capturing the inter -dependenci es among v arious labels is essential. Therefore, incor - porating RL can be an interesting research direction to e xplore and can potentially lead to no v el approaches and impro v ed ABSA models. 3. ABSA AS A MARK O V DECISION PR OCESS W e suppose a re vie w x is associated with n label triples of ( a i e , a i a , a i p ) ( i = 1 ...n ) which in v olv e n sentiment polarity a i p to w ards tar geted entities a i e and their associated aspects a i a . This can be modeled as a step-by-step decision-making process as sho wn in Figure 2: i) Step 1. Initialization: start wit h all states initialized with the input te xt x where no entities, aspects, or sentiments are identied, and x is yet to be processed. ii) Step 2. Sequential decision-making for the i th triple – Entity identication : at the state s t +1 , the agent scans x and identies the i th entity a i e (e.g., ‘hotel’). Update the state to s t +2 to include a i e , where t = ( i − 1) ∗ 4 . – Aspect identi cation : based on the current state s t +2 , the agent identies an aspect a i a related to the entity a i e (e.g., ‘cleanliness’ of ‘hotel’). Update the state to s t +3 to include a i e , a i a . – Sentiment pr ediction : the agent predicts the sentiment a i p for the identied a i a (e.g., positive senti- ment for ‘cleanliness’). Update the state to s t +4 to include a i e , a i a , a i p . iii) Step 3. Decision on ne xt triple prediction: the agent mo v es to the ne xt part of the te xt to identify more en- tities, aspects, and sentiments by repeating Step 2 - Sequential decision-making for the ( i + 1) th triple. It updates the ne xt state for the ne xt round of the sequential decision-making process until the entire te xt is processed. Otherwise, it reaches the ending state s n × 4 . i v) Step 4. Re w ard assignment: the agent recei v es re w ards based on the accurac y of its predictions. Cu- mulati v e re w ards guide the learning process, encouraging the agent to rene its polic y for better future predictions. Figure 2. The visualization of ABSA as a Mark o v decision process 4. MA THEMA TICALL Y DESIGN AN RL A GENT T O SOL VE ABSA The RL agent is represented as a tuple ( S , A , Y , q , r , γ ) , where S refers to the set of states, A denotes the set of nite possible actions (i.e., |A| < ∞ ), Y is the ground truth labels, q is transition probabilities by taking an action in a gi v en state, r is re w ard function, and γ ∈ [0 , 1] is a discount f actor . A deterministic curricular polic y π : S → A is a mapping from states to actions as follo ws: i) State: the state s ∼ S is represented as a tuple ( f , h ) , where f is a v ector representation of the re vie w x from input dataset D ; and h is an action history . In this study , x is represented by a list of consecuti v e A r einfor cement learning par adigm for V ietnamese aspect-based sentiment analysis (V iet The Bui) Evaluation Warning : The document was created with Spire.PDF for Python.
3378 ❒ ISSN: 2252-8938 tok en indices tok enized by a sub-w ord tok enizer (see Figure 3). This tok enizer is based on SentencePiece with a certain number of sub-w ord tok en strings, con v erts tok ens strings to ids and back, and adds se v eral special tok en strings to format the input re vie w such as: [CLS] - add to the be gin of the sentence to represent the whole tok ens; [UNK] - represent unkno wn tok ens which are not tok enized by tok enizer with the gi v en v ocab ulary; and [P AD] - e xtend padding tok ens to the list of tok en strings to mak e all input re vie ws ha v e the same length. Re vie w f is forw arded into a pre-trained transformer -based model B θ (e.g., BER T , ELECTRA, etc.), where θ refers to trained parameters, for e xtracting features. After that, we use the output hidden v ect or of tok en [CLS] as a representation v ector H ( f , θ ) = B θ ( f ) [ C LS ] ∈ R d B θ , where θ denotes pre-trained parameters, and d B θ is the hidden output dimension of B θ . ii) Action: the agent learns to predict a set of tuple of entity-aspect-polarity action A = { ( a i e , a i a , a i p ) } i =1 ..n , where a i e ∈ A e , a i a ∈ A a , and a i p ∈ A p are entity , aspect, and polarity actions respecti v ely; n is the number of predicted actions. Due to the hierarchical structure of tar get labels, each action is predicted by the current state s and action history h . Therefore, gi v en by a polic y model π , sequentially we predict an entity rst, an aspect ne xt, and a polarity last as sho wn in Figure 2. a e ∼ π ( H s , h | a e ); h ′ ← h ∪ { a e } a a ∼ π ( H s , h ′ | a a ); h ′′ ← h ′ ∪ { a a } a p ∼ π ( H s , h ′′ | a p ); h ′′′ ← h ′′ ∪ { a p } W e also add another binary step a t ∈ A t = { 0 , 1 } ( t stands for terminate) after predicting each tuple action ( a e , a a , a p ) to detect whether this is the last item of A or not. W e dene a t as 1 if the agent w ants to predict another tuple action, and as 0 otherwise. Basically , our polic y aims to predict consequence tuples ( a i e , a i a , a i p , a i t ) for solving hierarchical multi-label ABSA classication: ( a 1 e , a 1 a , a 1 p , a 1 t ) , ( a 2 e , a 2 a , a 2 p , a 2 t ) , ..., ( a n e , a n a , a n p , a n t ) where n is the number of predicted triples. iii) T ransitions: as described by the action prediction process, each episode (trajectory) has 4 × n steps. Each step is represented as a tuple ( f , h ) , and after taking an action a , the ne xt state becomes ( f , h ′ ) , where h ′ = h ∪ { a } . i v) Re w ard: we combine tw o types of re w ards into one re w ard r t at step t : r t = δ t I [ a t t == 1] r 1 ,t + (1 − δ t ) I [ T == 0] r 2 ,t Where r 1 ,t and r 2 ,t denotes dense e xploration and sparse competition re w ards respecti v ely; δ t is a linear annealing f actor . Our tar get is nding an optimal polic y π that maximizes the discounted long-term re w ard, which means the higher re w ard is the bet ter score. The dense e xploration re w ard r 1 ,t is calculated by the number of correct labels and the sparse competition re w ard r 2 ,t is calculated at the end of each episode τ by using a F 1 -score metric with a pair of predicted actions A τ and its tar gets Y τ : r 2 ,t = F 1 ( A τ , Y τ ) = 2 P r e − 1 ( A τ , Y τ ) + R ec − 1 ( A τ , Y τ ) ∈ R where P r e denotes precision score and R ec denotes recall score. In summary , we treat ABSA problem as an Mark o v decision process task where the agent learns to predict a list of consecuti v e actions and maximize the re w ard represe n t ed by F 1 -score. If the number of transitions is too small or too lar ge, the F P and F N might be much bigger than T P , the re w ard will be smaller , and the agent could be w orse. This en vironment is challenging to mak e the F 1 -score higher for e v ery incremental timestep. 5. DEEP Q-LEARNING FOR ABSA In this section, we delv e into the architecture and methodology of our deep Q-netw ork (DQN) model, which is tailored to address the comple x requirements of the tasks. As depicted in Figure 3, the architecture be gins by processing the input re vie w through a pre-trained transformer model, such as BER T or ELECTRA, Int J Artif Intell, V ol. 14, No. 4, August 2025: 3375–3385 Evaluation Warning : The document was created with Spire.PDF for Python.
Int J Artif Intell ISSN: 2252-8938 ❒ 3379 to e xtract a comprehensi v e and conte xtualized representation of t h e re vie w . This representation serv es as the input for the DQN, which acts as an agent learning to predict a sequence of actions necessary for na vig ating the hierarchical structure of the ABSA task. During the training phase, the primary focus is on rening the parameters of the Q-netw ork, ensuring that it can ef fecti v ely learn from the data and impro v e its predicti v e capabilities. Clean cheap room with enthusiastic staf f but a bit far from the center Clean cheap room ... center T rm T rm T rm T rm T rm T rm [CLS] [SEP] T rm T rm T rm T rm T rm T rm FC Action history FC LOCA TIONGENERAL negative ROOMS#CLEANLINESS positive SER VICE#GENERAL positive ROOMS#PRICES positive ... ... ... Sentence representation Input sentence Deep Q-network Pretrained transformer-based models Figure 3. An o v ervie w of our model architecture using pre-trained transformer -based models as re vie w representation encoders and DQN as an agent to learn to predict a sequence of actions (labels) for solving multi-label te xt classication In the con v entional RL models, the agent selects one action from the entire action set at each step, of ten without constraints, which can lead to repetiti v e actions within an episode. Ho we v er , in the conte xt of ABSA, the label triples < e, a, p > for a gi v en re vie w must be unique, pre v enting duplication within an episode. T o adapt this requirement to the RL frame w ork, our model is designed to ensure that the agent tak es non-repetiti v e actions throughout both the training and test ing phases. W e manage duplication by emplo ying a three-le v el tree structure. The initial le v el comprises the root, succeeded by entities, their respecti v e aspects, and ending with their potential polarity . T ra v ersing from the root to the leaf will establish the triples of < e, a, p > . An y pre viously visited branch is block ed by setting the log-lik elihood to − inf , equating to a probability of 0 . T o maintain sequential labeli ng, we update the action set after each step to remo v e duplication and enforce the agent to predict entities rst, follo wed by aspects, and nally their polarity . In a deterministic en vironment, we simplify the problem by assuming a x ed relationship between actions and subsequent states. The goal is to train a polic y π that maximizes the e xpected discounted cumulati v e re w ard, also kno wn as the r eturn , from an y starting state s 0 . This is mathematically e xpressed as: R t 0 = ∞ X t = t 0 γ t − t 0 r t Where γ ∈ [0 , 1) is the discount f actor . The discount f actor γ ensures that the innite sum con v er ges by reducing the weight of future re w ards, thereby prioritizing immediate re w ards o v er distant ones. This is crucial in balancing the trade-of f between short-term and long-term re w ards. The core of DQN is the action-v alue function Q ∗ : S × A → R , which pro vides the e xpected return of taking action a in state s and follo wing the optimal polic y thereafter . The optimal polic y π ∗ can be deri v ed by selecting the action that maximizes this function: π ∗ ( s ) = ar g max a Q ∗ ( s, a ) Since the true Q ∗ is unkno wn, we use a neural netw ork to approximate it. The netw ork parameters θ are updated to minimize the dif ference between the predicted Q-v alues and the tar get Q-v alues, deri v ed from the Bellman equation: Q π ( s, a ) = E [ r + γ Q π ( s ′ , π ( s ′ )) | s, a ] A r einfor cement learning par adigm for V ietnamese aspect-based sentiment analysis (V iet The Bui) Evaluation Warning : The document was created with Spire.PDF for Python.
3380 ❒ ISSN: 2252-8938 F or the optimal Q-function Q ∗ , the Bellman optimality equation is: Q ∗ ( s, a ) = E h r + γ max a ′ Q ∗ ( s ′ , a ′ ) | s, a i In practice, the Q-learning update rule is applied to iterati v ely adjust the parameters θ by minimizing the temporal dif ference error . Instead of using mean squared error (MSE), which is sensiti v e to outliers, we use the Huber loss. The Huber loss is dened as: L ( θ ) = E ( s,a,s ′ ,r ) ∈D [ L θ ( δ )] where D is the replay b uf fer containing past e xperiences, and the loss function L θ ( δ ) is: L θ ( δ ) = ( 1 2 δ 2 ( θ ) if | δ ( θ ) | ≤ 1 | δ ( θ ) | − 1 2 otherwise The Huber loss beha v es quadratically for small errors (lik e MSE) and linearly for lar ge errors (lik e mean absolute error (MAE)), pro viding rob ustness to outlier s in the Q-v alue estimates. The update to the parameters θ during training is performed using stochastic gradient descent or its v ariants, such as Adam [23], by computing the gradient of the loss function with respect to θ : θ ← θ − α ∇ θ L ( θ ) Where α is the learning rate, controlling the step size of the update. This detailed formulation allo ws the DQN to ef fecti v ely learn an approximation of the optimal Q-function by iterati v ely minimizing the temporal dif ference error , thereby impro ving the polic y π to maximize the e xpected return. The deep Q-l earning algorithm for reinforced ABSA is detailed in Algorithm 1. Our model emplo ys a feed-forw ard neural netw ork designed to handle state transitions by taking as input the dif ference between the current state s t and the pre vious state s t − 1 , represented as ∆ s t . Algorithm 1 DQN for ABSA Input: En vironment env ; Q netw ork Q θ ; trainable parameters θ ; learning rate α ; replay b uf fer D . θ ′ ← θ ▷ Initialize V parameters while not co v erage do s t ← env .r eset () ▷ Reset en vironment f or each en vironment step do a t ← arg max a t Q θ ( s t , a t ) ▷ Sample action s t +1 ← env .step ( a t ) ▷ T ak e an action a t D ← D ∪ { ( s t , a t , R ( a t | s t ) , s t +1 ) } ▷ Append transition to b uf fer D s t ← s t +1 end f or f or each gradient step do θ ← θ − α ∇ θ L ( θ ) ▷ Update Q netw ork θ ′ ← τ θ + (1 − τ ) θ ′ end f or end while The neural netw ork outputs a v ector of Q -v alues, Q ( s, a ; θ ) , where θ represents the parameters of the neural netw ork. The number of outputs corresponds to the discrete action space, which has a total dimension gi v en by: |A| = |A entity | + |A aspect | + |A polarity | + |A termination | Where |A entity | , |A aspect | , |A polarity | , and |A termination | are t he sizes of the action sets for entities, aspects, polarity , and termination, respecti v ely . Int J Artif Intell, V ol. 14, No. 4, August 2025: 3375–3385 Evaluation Warning : The document was created with Spire.PDF for Python.
Int J Artif Intell ISSN: 2252-8938 ❒ 3381 The Q-v alues are computed as Q ( s, a ; θ ) = f (∆ s t ; θ ) , where f is the function approximated by the neural netw ork, mapping state dif ferences to Q-v al ues for each possible action a . T o e xtract m eaningful features from each input re vie w , we utilize a pre-trained model B θ as the feature representation model. This model processes the ra w input x of a re vie w and generates a feature v ector ϕ ( x ) : ϕ ( x ) = B θ ( x ) This feature v ector ϕ ( x ) serv es as the input to the neural netw ork, allo wing the Q-learning algorithm to le v erage pre-trained kno wledge for impro v ed performance in sentiment analysis tasks. The action selection in each state in v olv es choosing the action a ∗ that maximizes the Q-v alue: a ∗ = arg max a ∈A Q ( s, a ; θ ) This action selection strate gy ensures that the polic y π is aimed at maximizing the e xpected cumulati v e re w ard o v er time. By inte grating these mathematical formulations, the model ef fecti v ely learns to map state-action pairs to their e xpected returns, thereby optimizing the polic y for sentiment analysis tasks in a discrete action space. 6. EXPERIMENTS This section rst presents the dataset used to conduct e xperiments. Then, e xperimental setups such as e v aluation metrics and model training parameters are sho wn. Finally , e xperimental results are elaborated and compared with se v eral strong SO T A baselines. 6.1. Datasets The dataset used to perform e xperiments is a public dataset released in the ABSA challenges of VLSP 2018 at https://vlsp.or g.vn/vlsp2018/e v al/sa. The campaign dealt with data in tw o domains, namely restaurants and hotels. The number of entities, their aspects, and corresponding polarity mentioned in each re vie w are di v erse. The information about the training, de v elopment, and test sets is summarized in T able 1. F or each re vie w , its aspects are identied by the tuples (entities, attrib utes) (Phase A). While in Phase B, we need to further identify the polarity for each aspect tuple as positi v e, ne g ati v e, or neutral. T able 1. Statistics on the train, de v , and test sets of tw o datasets on tw o domains Domain Dataset #Re vie ws #Aspects Restaurant T raining 2961 9034 V alidation 1290 3408 T est 500 2419 Hotel T raining 3000 13948 V alidation 2000 7111 T est 600 2584 6.2. Experimental setups T o e v aluate the performance, we use the metrics proposed by the campaign. The y are precision, recal l, and F1-score in micro-a v eraged. In conducting e xperiments, we implemented the models using Pytorch. F or pre-training language models, we e xploited dif ferent ones a v ailable for V ietnamese. The y are mBER T [24], viBER T [25], phoBER T [26], and vElectra [25]. The models’ h yper -parameters were chosen via a search on the v alidation set. W e v aried dif ferent h yper -parameters to nd the optimized sets for ones such as lter windo w sizes, dropout rates, optimization methods, learning rates, batch sizes, and number of epochs. 6.3. Experimental r esults T w o kinds of e xperiments were implemented. First, we conducted e xperiments to v erify ho w pre- trained transformer -based models impact the prediction performance of the proposed RL-based approach. Then, we compare this proposed approach with the e xisting SO T A results on tw o benchmark datasets. A r einfor cement learning par adigm for V ietnamese aspect-based sentiment analysis (V iet The Bui) Evaluation Warning : The document was created with Spire.PDF for Python.
3382 ❒ ISSN: 2252-8938 6.3.1. The effecti v eness of differ ent pr e-trained language models on the RL-based appr oach Figures 4 and 5 dra w the con v er gence curv es for the RL approach using four a v ailable pre-trained language models on tw o phases. The y illustrate the con v er gence patterns observ ed in the hotel and restaurant datasets. It’ s e vident from these plots that the algorithms con v er ge ef fecti v ely after se v eral iterations on de v el- opment sets . Notably , emplo ying vElectra as a re vie w encoder resulted in the highest performance, follo wed by phoBert, viBER T , and mBert. 0 50 100 0.65 0.7 0.75 0.8 0.85 Number of epochs F1 score De v - Phase 1 0 50 100 0.65 0.7 0.75 0.8 Number of epochs F1 score De v - Phase 2 0 50 100 0.6 0.65 0.7 0.75 0.8 0.85 0.9 mBert ViBert vEelectr a phoBert Figure 4. Con v er gence curv es on the de v elopment set of the hotel dataset 0 50 100 0.75 0.8 0.85 Number of epochs F1 score De v - Phase 1 0 50 100 0.65 0.7 0.75 0.8 Number of epochs F1 score De v - Phase 2 0 50 100 0.6 0.65 0.7 0.75 0.8 0.85 0.9 mBert ViBert vEelectr a phoBert Figure 5. Con v er gence curv es on the de v elopment set of the restaurant dataset 6.3.2. Comparing the RL-based appr oach with the existing SO T A w ork This section presents the e xperimental results of the proposed approach compared to the SO T A results on tw o public V ietnamese datasets. T o conduct a comprehensi v e comparison, we’ v e chosen the best three e xist- ing methods: JointLearning-mBER T , JointLearning-viBER T , and MultiT ask-PhoBER T . F or the RL approach, we implemented the algorithm using four dif ferent backbone models. Each model utilizes a distinct pre-trained language model for V ietnamese: mBER T , viBER T , phoBER T , and vElectra. T ables 2 and 3 sho w e xperimental results on the test sets of tw o domains. F or the hotel domain, it can be seen from T able 2 that the RL approach performed better than all e x- isting models. Using vElectra and phoBER T yielded the best performance. Notably , vElectra slightly captures the nuances of re vie ws better than phoBER T , leading to impro v ed F1 scores in both phases. W e achie v e a ne w SO T A of 83.51% in the F1 s core for Phas e 1, and 78.55% in the F1 score for Phase 2. Using the best RL-vElectra model, we achie v ed an F1 score of 83.52% for Phase 1 and 78.55% for Phase 2 on the hotel dataset. F or the restaurant domain, the RL approach demonstrat ed notably superior performance compared to the three e xisting SO T A models (see T able 3). W e also see an impro v ement in the F1 scores in both tw o phases. Compared to the best model of MultiT ask-phoBER T , the RL-vElectra signicantly boosted the F1 scores by 1.02% for Phase 1, and 2.8% for Phase 2. Using the best RL-vElectra yielded the highest performance of 85.25% for Phase 1, and 74.35% for Phase 2 on the restaurant dataset. Using the same pre-trained language models, the proposed RL approach consistently outperformed the con v entional classication approach across both datasets. Notably higher performance w as achie v ed with both mBER T and viBER T , while phoBER T yielded slightly better results. Among all language models, using Int J Artif Intell, V ol. 14, No. 4, August 2025: 3375–3385 Evaluation Warning : The document was created with Spire.PDF for Python.
Int J Artif Intell ISSN: 2252-8938 ❒ 3383 v-Electra achie ving the best performance and established a ne w SO T A result on both tw o datasets. These ndings demonstrate that the proposed RL approach surpasses the e xisting SO T A classication methods. T able 2. Experimental results on the test set of the hotel domain Phase 1 Phase 2 Pre Rec F1 Pre Rec F1 Baselines JointLearning-mBert [25] 85.32 76.04 80.42 78.17 65.56 71.31 JointLearning-viBert [25] 83.93 80.26 82.06 80.04 70.01 74.69 MultiT ask-phoBER T [9] 87.45 78.17 82.55 81.90 73.22 77.32 Reinforcement Learning RL-mBert 85.96 77.38 81.45 79.67 71.13 75.16 RL-viBert 86.59 79.83 83.08 81.04 74.30 77.52 RL-phoBert 86.28 79.53 82.76 81.38 74.65 77.87 RL-vElectra 87.82 79.60 83.51 82.86 74.66 78.55 T able 3. Experimental results on the test set of the restaurant domain Phase 1 Phase 2 Pre Rec F1 Pre Rec F1 Baselines JointLearning-mBert [25] 84.03 82.64 83.33 65.70 71.43 68.45 JointLearning-viBert [25] 84.21 84.25 84.23 69.75 72.92 71.30 MultiT ask-phoBER T [9] - - 84.23 69.66 73.54 71.55 Reinforcement Learning RL-mBert 83.57 84.61 84.09 69.54 70.58 70.05 RL-viBert 84.35 84.46 84.41 72.73 72.84 72.78 RL-phoBert 84.13 84.46 84.30 72.86 73.19 73.02 RL-vElectra 85.89 84.62 85.25 74.99 73.72 74.35 7. CONCLUSION In this paper , we present a ne w and ef fecti v e approach to sentiment analysis for V ietnamese which le v erages RL techniques instead of con v entional approaches. Instead of modelling the tasks as a classication problem with supervised learning techniques, we formulate the prediction task as a Mark o v decision process. Here, we deplo y a RL agent t o sequentially predict labels, ef fecti v ely le v eraging re vie w features and pre viously predicted labels. The agent seeks to disco v er optimal policies by maximizing long-term re w ards, reecting pre- diction accurac y . The e xperimental results on public V ietnamese datasets sho wed that the proposed approach yielded ne w SO T A results in both tw o domains of hotels and restaurants. Specically , we achie v ed 83.51% in the F1 score for Phase 1, and 78.55% in the F1 score for Phase 2 on the hotel dataset. Emplo ying the identical RL-vElectra model lik e wise achie v ed the highest performance, reaching 85.25% for Phase 1 and 74.35% for Phase 2 on the restaurant dataset. In the future, we continue to try dif ferent types of re w ard functions to nd the best one. Other kinds of lar ge language models should also be inte grated to optimize the re vie w representation that captures the full conte xt of re vie ws. FUNDING INFORMA TION This paper is funded by the International School, V ietnam National Uni v ersity Hanoi (VNU-IS) under the local project CS.2023-11. A UTHOR CONTRIB UTIONS ST A TEMENT This journal uses the Contrib utor Roles T axonomy (CRediT) to recognize indi vidual author contri b utions, reduce authorship disputes, and f acilitate collaboration. A r einfor cement learning par adigm for V ietnamese aspect-based sentiment analysis (V iet The Bui) Evaluation Warning : The document was created with Spire.PDF for Python.
3384 ❒ ISSN: 2252-8938 Name of A uthor C M So V a F o I R D O E V i Su P Fu V iet The Bui ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Linh Thuy Ngo ✓ ✓ ✓ ✓ ✓ ✓ ✓ Oanh Thi T ran ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ C : C onceptualization I : I n v estig ation V i : V i sualization M : M ethodology R : R esources Su : Su pervision So : So ftw are D : D ata Curation P : P roject Administration V a : V a lidation O : Writing - O riginal Draft Fu : Fu nding Acquisition F o : F o rmal Analysis E : Writing - Re vie w & E diting CONFLICT OF INTEREST ST A TEMENT Authors state no conict of interest. D A T A A V AILABILITY The data that support the ndings of this study are a v ailable from the competition VLSP 2018 - Aspect Based Sentiment Analysis. Data are a v ailable at https://vlsp.or g.vn/vlsp2018/e v al/sa with the permission of VLSP representati v es. REFERENCES [1] D. V . Thin, V . D. Nguye, K. V . Nguyen, and N. L.-T . Nguyen, “Deep learning for aspect detection on V ietnamese re vie ws, ” in 2018 5th N AFOSTED Confer ence on Information and Computer Science (NICS) , IEEE, No v . 2018, pp. 104–109, doi: 10.1109/NICS.2018.8606857. [2] H.-Q. Nguyen and Q.-U. Nguyen, “ An ensemble of shallo w and deep learning algorithms for V ietnamese sentiment analy- sis, ” in 2018 5th N AFOSTED Confer ence on Information and Computer Science (NICS) , IEEE, No v . 2018, pp. 165–170, doi: 10.1109/NICS.2018.8606880. [3] J. Meng, Y . Long, Y . Y u, D. Zhao, and S. Liu, “Cross-domain te xt sentiment analysis based on CNN FT method, ” Information , v ol. 10, no. 5, May 2019, doi: 10.3390/info10050162. [4] N. C. Le, N. T . Lam, S. H. Nguyen, and D. T . Nguyen, “On V ietnamese sentiment analysis: a transfer learning method, ” in 2020 RIVF Int ernational Confer ence on Computing and Communication T ec hnolo gies (RIVF) , IEEE, Oct. 2020, pp. 1–5, doi: 10.1109/RIVF48685.2020.9140757. [5] M . Singh, A. K. Jakhar , and S. P ande y , “Sentiment analysis on the impact of corona virus in social life using the BER T model, ” Social Network Analysis and Mining , v ol. 11, no. 1, Dec. 2021, doi: 10.1007/s13278-021-00737-z. [6] O. T . T ran and V . T . Bui, “ A BER T -based Hierarchical model for V ietnamese aspect based sentiment analysis, ” in 2020 12th International Confer ence on Knowledg e and Systems Engineering (KSE) , IEEE, No v . 2020, pp. 269–274, doi: 10.1109/KSE50997.2020.9287650. [7] M . Zouidine and M. Khalil, “ Arabic sentiment analysis based on deep reinforcement learning, ” in 2022 5th International Confer ence on Networking , Information Systems and Security: En visa g e Intellig ent Systems i n 5g//6G-based Inter connected Digital W orlds (NISS) , IEEE, Mar . 2022, pp. 1–5, doi: 10.1109/NISS55057.2022.10085147. [8] M. E. Basiri, S. Nemati, M. Abdar , E. Cambria, and U. R. Acharya, “ ABCDM: An attention-based bidirectional CNN- RNN deep model for sentiment analysis, ” Futur e Gener ation Computer Systems , v ol. 115, pp. 279–294, Feb . 2021, doi: 10.1016/j.future.2020.08.005. [9] H.-Q. Dang, D.-D.-A. Nguyen, and T .-H. Do, “Multi-task solution for aspect cate gory sentiment analysis on V ietname se datasets, ” in 2022 IEEE International Confer ence on Cybernetics and Computational Intellig ence (CyberneticsCom) , IEEE, Jun. 2022, pp. 404–409, doi: 10.1109/CyberneticsCom55287.2022.9865479. [10] R. S. Sutton and A. G. Barto, Reinfor cement learning: an intr oduction , Cambridge, Massachusetts: MIT Press, 1998. [11] P . Abbeel, A. Coates, M. Quigle y , and A. Y . Ng, “ An application of reinforcement learning to aerobatic Helicopter ight, ” in NIPS 2006: Pr oceedings of the 19th International Confer ence on Neur al Information Pr ocessing Systems , in NIPS’06. Cambridge, MA, USA: MIT Press, 2006, pp. 1–8, doi: 10.7551/mitpress/7503.003.0006. [12] S. W ang, H. Liu, P . H. Gomes, and B. Krishnam achari, “Deep reinforcement learning for dynamic multichannel access in wire- less netw orks, ” IEEE T r ansactions on Co gnitive Communications and Networking , v ol. 4, no. 2, pp. 257–265, Jun. 2018, doi: 10.1109/TCCN.2018.2809722. [13] Y . Y ifei and S. Lakshminarayanan, “Multi-agent reinforcement learning for process control: Exploring the intersection between elds of reinforcement learning, control theory , and g ame theory , ” The Canadian J ournal of Chemical Engineering , v ol. 101, no. 11, pp. 6227–6239, No v . 2023, doi: 10.1002/cjce.24878. [14] V . Uc-Cetina, N. Na v arro-Guerrero, A. Martin-Gonzalez, C. W eber , and S. W ermter , “Surv e y on reinforcement learning for language processing, ” Articial Intellig ence Re vie w , v ol. 56, no. 2, pp. 1543–1575, Feb . 2023, doi: 10.1007/s10462-022-10205-5. [15] M. W ankhade, A. C. S. Rao, and C. K ulkarni, “ A surv e y on sentiment analysis methods, applications, and challenges, ” Articial Intellig ence Re vie w , v ol. 55, no. 7, pp. 5731–5780, Oct. 2022, doi: 10.1007/s10462-022-10144-1. Int J Artif Intell, V ol. 14, No. 4, August 2025: 3375–3385 Evaluation Warning : The document was created with Spire.PDF for Python.