Indonesian J our nal of Electrical Engineering and Computer Science V ol. 38, No. 2, May 2025, pp. 1353 ∼ 1366 ISSN: 2502-4752, DOI: 10.11591/ijeecs.v38.i2.pp1353-1366 ❒ 1353 Indonesian sentiment analysis in natural en vir onment topics Christofer Octo vianto 1 , Muhammad Okk y Ibr ohim 1,2 , Indra Budi 1 1 F aculty of Computer Science, Uni v ersitas Indonesia, Depok, Indonesia 2 Dipartimento di Informatica, Uni v ersit ` a De gli Studi di T orino, T urin, Italy Article Inf o Article history: Recei v ed Apr 6, 2024 Re vised No v 12, 2024 Accepted No v 24, 2024 K eyw ords: Data-dri v en analyses IndoBER T model Instagram Natural en vironment topic Sentiment analysis ABSTRA CT Indonesia is one of the countries that is rich in biodi v ersity and has a high pop- ulation gro wth. This condition can ca use Indonesia to ha v e problems related to the natural en vironment that are more comple x than other countries. Hence, this has created a lot of discussions re g arding natural en vironmental issues in Indone- sia on social media platforms. In this case, stak eholders lik e the go v ernment in general can utilize sentiment analysis (SA) to comprehend the public’ s vie ws to allo w them to better t the public’ s e xpectations when formulating a particular polic y that related to the en vironmental s ustainability (ES) issues. This paper b uilt the rst open dataset of Indonesian SA dataset in ES topics collected from Instagram. As the benchmark of our dataset, we used IndoBER T model v ariant for constructing the model and the e xperiment result sho ws that model based on IndoBER T -lar ge-p2 obtained the best performance with 72.44% of F1-score. This is an open access article under the CC BY -SA license . Corresponding A uthor: Muhammad Okk y Ibrohim F aculty of Computer Science, Uni v ersitas Indonesia Depok, Indonesia Email: okk yibrohim@cs.ui.ac.id, muhammadokk y .ibrohim@unito.it 1. INTR ODUCTION En vironmental sustainability (ES) is one topic that has recei v ed more attention in this era. This t ak es place mainly because its inuence all aspects of life especially in health and life quality . The issues in this domain must be discussed and solv ed together collecti v ely by the go v ernments and the public. Go v ernments ha v e more crucial role since the y can propose and control the ES’ s polici es and re gulations. The Public also has an essential role in terms of supervising and e v aluating the go v ernment’ s policies and actions, and reporting to their go v ernment if there are ES issues around them. In the current social media era, citizens often discuss and report an ES issue on their social media. Hence, the relationship between people and the en vironment through sentiment analysis (SA) may be crucial to be e xplored [1]. SA has been broadly applied to s e v e ral topics SA on patient feedback [2], [3], public f acility re vie ws [4], [5], and product re vie ws [6], [7], to g auge ho w people percei v e services, products, or other specic dis- course topics. Concerning en vironmental issues, SA can help us comprehend the public’ s vie ws of the condition of the en vironment and the go v ernment en vironmental policies that are happening. This can allo w go v ernments to better t the public’ s e xpectations when the y decide on what needs to be x ed for en vironmental issues in their representati v e. In recent years, some research has been done to e xplore SA applied to ES topics from v ari o us per - specti v es with v arious approaches. Ibrohim et al. [1], a systematic re vie w of SA for ES topics is conducted to e xplore what tasks, techniques, and benchmarks ha v e been used in this gro wing research area. Dahbi et al. J ournal homepage: http://ijeecs.iaescor e .com Evaluation Warning : The document was created with Spire.PDF for Python.
1354 ❒ ISSN: 2502-4752 [8] ha v e researched SA in the Moroccan language related to the smart city in Morocco. Zhang et al. [9] con- ducted sentiment classication research related to public opinions about food safety in China. The y used the IF oodCloud application to g ather public opinions and used machine learning and deep learning algorithms to classify the data. Indonesia is the country with the second most biodi v erse country [10] and the fourth lar gest population [11]. This condition can cause Indonesia to ha v e problems relat ed to t he ES that are more comple x than other countries because there will be a lot of interaction between humans and the natural en vironment components. This situation has also created a lot of discussions re g arding natural en vironmental issues in Indonesia in ne wspapers, blogs, and social media. On social media, users eng age in online interaction which leads to the generation of a substantial v olume of user -generated information. In addition, social media users come from all o v er the w orld with dif ferent interests and mindsets [12]. The de v elopment of this v ast user -generated content can impro v e the management of e xisting ES monitoring systems. Based on surv e y conducted by [13], Instagram is the most used social media for discussing and campaigning on natural en vironment issues (especially climate change) among users with an age range 20-29 years old, which is the age range of users with the most social media e xposure in Indonesia. Therefore, we can get a wide v ariety of information, specically in this study for collecting data and analyze the sentiment with re g ards to the ES issues in Indonesia. T o the best of our kno wledge, research for SA applied in ES related topics in Indonesia n social media is still v ery rare. Michael and Utama [14] used Nai v e Bayes (NB) for conducting SA in w aste management cases by utilizing Indonesian tweets. Indra et al. [15] used the k-nearest neighbor (KNN) algorithm to comprehend the public vie w of social and political orientation in Pekanbaru Ci ty by utilizing Indonesian tweets. One of the social orientations used is en vironmental issues in Pekanbaru City . Sugiharti and F auziah [16] used KNN and maximum entrop y methods and Indonesian tweets to g auge public vie ws on menstrual cups to reduce menstrual w aste. F ontanella et al. [17] used NL TK to analyze the sentiment of the public in Indonesian tweets concerning the implementation of en vironmental protection and management of coal w aste. F or research in Indonesian SA about ES topics, unfortunately , we found that all pre vious w orks do not share their dataset with the public so other researchers cannot reproduce and enhance their research. In the terms of classier used, we found that most of the algorithms pre viously mentioned rely on hea vy feature engineering and pose challenges in terms of optimiza tion. SA, especiall y using tradi tional ma- chine learning algorithms, f ails to capture the conte xt or the meaning in entire sentences of tweet data due to a limited understanding of semantics and w ord sequence as the y only see certain patterns. This situation requires additional preprocessing or feature engineering. Therefore, this mak es analysis more dif cult for an algorithm to accurately capture the intended meaning and conte xt of a sentence [18], [19]. No w adays, most of the state-of-the-art of SA tasks ha v e been achie v ed by using a model based on bidirectional encoder representations from transformers (BER T) pre-trained model [20]–[23]. In this paper , we focus on the implementation of Indonesian BER T (IndoBER T) [24]. In summary , the main contrib utions of our paper are as follo ws: − Building the rst open dataset and model f o r ES vs non-ES topic classication that is useful for ES data ltering before SA annotation; − Building the rst open dataset for Indonesian Instagram SA in SE topics where each data is annotated by three nati v e annotators; − Pro viding benchmark e xperi ment for our dataset including error analysis to help the community in conduct- ing a good system for ES monitoring. W e release our code and dataset in [25]. The rest of this paper is or g anize d as follo ws. Section 2 discusses related w orks, section 3 discusses the step for b uilding the dataset and e xplaining ho w our e xperi- ments are conducted , section 4 presents the e xperiment results and analysis. Finally , section 5 discusses the conclusions and future w orks suggestions of this paper . 2. RELA TED W ORKS Studies related to SA in ES topics ha v e increased in the past fe w years. Dahbi et al. [8] conducted SA using T witter and F acebook data in the Moroccan language re g arding the smart cities in Morocco. The characteristics of smart ci ty in the dataset include pollution, congestion, security risks, and urban resources. The y used support v ector machine (SVM), NB, KNN, and decision tree (DT) algorithms to b uild the SA model. Binary weighting for feature e xtraction is used in this research. The highest accurac y v alue w as achi e v ed by Indonesian J Elec Eng & Comp Sci, V ol. 38, No. 2, May 2025: 1353–1366 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1355 using the SVM algorithm with an accurac y of 94.1%. Zhang et al. [9], conducted SA about public opinions of food safety in China such as vie ws on food adulteration, food-borne diseases, agricultural pollution, irre gular food distrib ution, and problems with food production. The data used w as tak en from IF oodCloud, a platform b uilt for real-time SA of public opinion on food safety in China. The y collect about 3,100 data for e xperiments and b uilt a model us ing the le xicon-based algorithm, SVM, multilayer perceptron (MLP), and long short-term memory (LSTM). The best model is achie v ed by the LSTM algorithm with 97.37% of F1-score. In Indonesia, there also are studies in v olving SA on ES topics. Mic h a el and Utama [14], conducted SA concerning w aste management cases in Indonesia with the help of the KNN algorithm. The cases were collected from Indonesia tweets with the k e yw ords of sampah (w aste), polusi (pollution), lingkungan (en viron- ment), and manajemen (management). The KNN model achie v ed an accurac y of 65.02% for this task. Indra et al. [15] utilized the KNN algorithm to comprehend the social ori entation of the public in Pekanbaru city . One of the public orientations discussed w as en vironmental issues. The data were collected from Indonesian tweets. En vironmental issues were the social orientation component that recei v ed the most ne g at i v e sentiment (56% of the en vironmental data collected). Based on the discourses, “ sampah ” (g arbage, w aste, and trash) w as the most posted and discussed w ord on the data along with “ banjir ” (ood) and “ polusi udar a ” (air pollution). Sugiharti and F auziah [16] b uilt an SA model based on maximum entrop y and KNN algorithms to comprehend public vie ws on menstrual w aste and the utilization of menstrual cups. The y used 1,108 Indonesian tweets for b uilding the model. The best model w as achie v ed based on maximum entrop y with an accurac y of 84.6%. F ontanella et al. [17] used NL TK to analyze the sentiment of the public concerning the implementation of en- vironmental protection a nd management of coal w aste. The y collected data from Indonesian tweets with a total of 236 tweets. The results sho wed that the public opinions tended to be ne g ati v e sentiments with an accurac y rate is 77.40%. From our literature re vie w process and also stated in a surv e y by [1] ndings, there is one big issue for the NLP communities in b uilding an SA model in ES topics, i.e. the e xistence of an open dataset. Especially for the Indon e sian dataset, all pre vious research only w orks in small datasets in a specic ES t opic where the y do not share the dataset with the public. Therefore, as a country that potentially has bigger ES issues than other countries as mentioned in section 1, there is a need to b uild an open dataset that widely co v ers ES topics issue in this language so that the community can help the Indonesian go v ernment in conducting a good ES monitoring system. In 2023, Bosco et al. [26] created a complete annotation guideline for collecting and annotating structured s entiment analysis (SSA) data in ES topics. The y used 12 k e y w ords to collect Italian T wi tter data and 120 k e yw ords to collect 120 English T witter data. The dataset collected co v ers all 10 ES topics dened by [1] nam ely sustainability , en vironment, green, or g anism, pollution, w aste, food, ener gy , carbon, and climate change. F or the annotation process, the y annotate their dataset in tw o schemas. F or the Italian dataset, the y perform complete SSA annotation where a tweet is annotated in span and relation tuple label (holder , tar get, sentiment term, sentiment polarity , and topic). Meanwhile, for the English dataset, the y annotate the sentiment term and its polarity in the form of relation tuple (sentiment term and sentiment polarity) and plan to continue to get full annotation scheme as the y did in the Italian data in the second stage. In this case, both Italian and English datasets are annotated by three annotators for each data, where the y use a nati v e e xpert annotator for Italian data and use nati v e cro wdsource annotator from Prolic [27] for English data. F or the e v aluation, the y use Fleiss’ [28] and Cohen’ s κ [29] score and sho w that both datasets the y b uilt are v alid for the e xperiment. Unfortunately , their w ork is only focused on dataset b uilding without e xperimenting. The y also do not discuss ho w to aggre g ate the nal label as the y annotate at span-le v el us ing more than one annotator . F or span-le v el aggre g ation, Barbieri et al. [30] annotated the SA dataset with the same schema did by [26] in their English dataset. In this case, Barbieri et al. [30] con v ert the span-le v el annotation to the document-le v el annotation via majority v oting between annotated sentiment terms. As m entioned in section 1 and also stated in surv e y ndings conducted by [1], most of the e xisting SA in ES topics modeling still rely on le xicon-based libraries and classic machine learning algorithms. Meanwhile, the current state-of-the-art in SA modeling relies on a BER T -based pre-trained model. F or SA in Indonesian data, W ilie et al . [24] sho ws that the BER T -based model outperforms all classic machine learning and classic deep learning models in the SA task in the general topic. In this case, the y train v arious BER T -based models named IndoBER T and share the pre-trained model with the public so that the NLP community can ne-tune it for a specic do wnstream task. Therefore, it is interesting to us e this IndoBER T -based model for Indonesian SA in ES topics modeling as a benchmark. Indonesian sentiment analysis in natur al en vir onment topics (Christofer Octo vianto) Evaluation Warning : The document was created with Spire.PDF for Python.
1356 ❒ ISSN: 2502-4752 3. METHOD This section details the process of applying SA for the natural en vironment using IndoBER T -based pre-trained models. The o w of our research e xperiment can be seen in Figure 1. In general, the process is di vided into tw o, namely dataset b uilding as (subsection 3.1) and model b uilding (subsection 3.2). Figure 1. Research e xperiment o w 3.1. Dataset b uilding The data used for creating the dataset for this paper comes from Instagram. Data in the form of an Instagram post caption is collected with InstaT ouch [31] library within the date range of 9 August 2023 until 10 October 2023. W e translate and e xtend k e yw ords used by [26] and obtained 152 k e yw ords in total [25]. In the data collection process, we use both w ord/phrase form and hashtag form as done by [26]. From this process, the amount of Instagram data that w as successfully collected w as 135,007 data (including some duplicate data in which more than one of the k e yw ords occurred). T o get a v alid dataset, we perform data ltering before the data annotation process. First, we erased duplicate and non-Indonesian data using Langdetect library [32] to a v oid e v aluation bias result i n the modeling process later . This process gi v es 37,158 unique Indonesian data. Second, we ltered each data based on whether its content is on ES topics or not to a v oid non-ES data contamination in our dataset. Based on preliminary qualitati v e analysis, there is some data that its content is not on en vironmental topics b ut contains the k e yw ords that we used to collect Instagram data. This data instead discusses adv ertising, promotions, and lifestyle. Therefore, we created a model that can lter Instagram data that has conte xt on en vironmental topics or not. T o the best of our kno wledge, there has been no research in Indonesian ES topic classication. W e b uilt a dataset for this task with a manually labeled sample of 600 data from pre vious ltering results with the composition of 300 data for ES topic and 300 data for non-ES topic. Then, we split the sample with a sta nd a rd split ratio of 80% for training data and 20% for testing data. W e used one of IndoBER T -based models namely IndoBER T - base-p1 [24] and achie v ed a macro F1-score o f 87.60%. This result indicates the model is good enough to be used for en vironmental topic ltering. W e do not do man y e xperiments for this process as this is only used for ltering before the manual annotation process. 8,889 data ha v e content on ES topics from this ltering process. Third, we ltered based on automatic sentiment label to a v oid e xtreme data-imbalanced in re g ards to the manual sentiment labeling process label. W e used the Akahana model [33], an IndoBER T -based model that already ne-tuned for Indonesian sentiment classication at the document le v el. T o k eep balancedness data across ES topics, we tar get 500 data for each topic with the com position of 175 with positi v e sentiment, 150 data with neutral sentiment, and 175 data with ne g ati v e sentiment from the automatic sentiment labeling process . From 10 ES t opics , only v e topics achie v ed this tar get namely climate change, ener gy , en vironment, pollution, and w aste, so we obtained 2,500 Instagram posts for the manual annotation process. F or the annotation par t, we adapted annotation data guidelines and process from [26] for their En- glish dataset . W e annotate e xpression terms (span-le v el annotation) which are groups of adjacent or closely connected w ords in the data that contain a positi v e e xpression (labeled as Exp Positive ) or ne g ati v e e x- pression (labeled as Exp Negative ). The positi v e e xpression represents an act of supporting, gi ving a good assessment, and appreciating an action or polic y related to the natural en vironment. Meanwhile, the ne g a- Indonesian J Elec Eng & Comp Sci, V ol. 38, No. 2, May 2025: 1353–1366 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1357 ti v e e xpression represents rejecting, gi ving a ba d assessment, and criticizing an action or polic y related to the natural en vironment. W e hired three nati v e Indonesian speak ers that ha v e at least a bachelor’ s de gree for this task. The Langing Annotate (an online data annotation platform) [34] is us ed to annotate the data. T o ensure that all three annotators understood the annotation guidelines, we conducted a pilot study by asking them to annotate 150 data samples that ha v e been annotated by us and e v aluate the the agreement score between each annotator and with us using pairwise Cohen’ s κ [29]. If an annotator candidate has an a v erage pairwise Cohen’ s κ less than 0.4, we will look for ne w annotator candidate and test them with the same procedure. W e follo w the procedure of κ interpretation le v el gi v en by [35] for this e xamination. Once the a v erage pairwise Cohen’ s κ for all annotators is more than or equal to 0.4, we ask them to annotate all 2,500 data that we obtained from the ltration process. When the annotators ha v e completed their annotations, we calculated Fleiss’ κ [28] to e v aluate the agreement le v el for all annotator results and we got a score of 0.5583, indicating the agreement le v el has a moderate le v el [35] and is suitable for the e xperiment. The e xample of annotation results can be seen in Figure 2. In Figure 2(a), there are tw o terms an- notated as ne g ati v e e xpressions. The rst term “ harus se g er a dihentikan ” (must be stopped immediately) represents rejection of industry that produces carbon emission. The second term “ ber dampak r elatif lebih besar terhadap pencemar an lingkungan ” (has a relati v ely greater impact on en vironmental pollution) repre- sents criticism of carbon emission produced by industry around the city that leads to en vironmental pollution. In Figure 2(b) there are tw o terms annotated as ne g ati v e e xpressions and one term annotated as positi v e e xpression. The rst term “ bikin cadangan ener ginya makin sedikit ” (reduce ener gy reserv es) and the second term “ dampak b uruknya ” (the bad impact) represent criticism of the impact of continuous use of fossil ener gy . The third term “ gak akan habis ” (will not run out) represents gi ving a good assessment of using rene w able ener gy . In Figure 2(c), there are tw o terms annotated as positi v e e xpressions. The rst term “ langkah k ecil menuju perubahan besar ” (small steps to w ards big change) and the second term ” mencapai masa depan yang lebih cer ah dan berk elanjutan ” (achie v e a brighter and more sustainable future) represent an act of supporting the use of rene w able ener gy , smart w aste processing, and emission reduction. In Figure 2(d), there are tw o terms annotated as positi v e e xpressions and one term annotated as ne g- ati v e e xpression. The rst term “ setorkan sampah terpilah k e bank s amp ah ter dekat ” (deposit sorted w aste to the nearest w aste bank) and the second term “ terinte gr asi ” (inte grated) represent support and benets for the use of w aste banks in the local community , respecti v ely . The third term “ mencemari lingkungan ” (contaminate the en vironment) represents the ne g ati v e impacts that can occur if w aste is not managed properly in the local community . In Figure 2(e), there is no term annotated as either ne g ati v e or positi v e e xpression. In Figure 2(f), there are equally terms annotated as positi v e and ne g at i v e e xpressions. The rst term “ teng g elam ” (dro wned) and “ se gala bahayanya ” (all the dangers) are directed to the ne g ati v e ef fects about w aste felt in the local com- munity . The third term “Y uk sama-sama” (let’ s do it together) and the fourth term “ mengur angi sampah plastik ” (reduce the plastic w aste) are directed to the local community to support plastic w aste reduction mo v ement. W e use majority v oting in code to determine the nal label for each data in tw o steps follo wi n g [30] . First, it con v erts span-le v el labels to document-le v el labels by performing majority v oting on each annota- tor’ s res ults. Majority v oting is performed by calc ulating the number of highes t e xpression terms: labeled as “POSITIVE” if the highest e xpression term is a positi v e e xpression and labeled as “NEGA TIVE” if the highest e xpression term is a ne g ati v e e xpression. If the number of terms containing ne g ati v e e xpressions and positi v e e xpressions is the same or there are no e xpression terms at all in the data, then the data will be labeled as “NEUTRAL”. The sample of con v ersion result to document-le v el label can be seen in T able 1. Second, the code determines the nal label by performing majority v oting on the three document-le v el labels on each data. This process occurs because we ha v e three document-le v el labels from three annotator results for each data, b ut the model we will use requires only one label as input for each data. There is se v eral data is remo v ed because no agreement is found (strong disagreement). Therefore, the total data in t he dataset for this e xperiment is 2,391 data which consists of 1,251 data with the label “POSITIVE”, 288 data with the label “NEUTRAL”, and 852 data with the label “NEGA TIVE”. Indonesian sentiment analysis in natur al en vir onment topics (Christofer Octo vianto) Evaluation Warning : The document was created with Spire.PDF for Python.
1358 ❒ ISSN: 2502-4752 (a) (b) (c) (d) (e) (f) Figure 2. Example of data with span-le v el labels: (a) ne g ati v e e xpression term, (b) majority ne g ati v e e xpression term, (c) positi v e e xpression term, (d) majority positi v e e xpression term, (e) no e xpression term, and (f) equal e xpression term T able 1. Example of document-le v el label con v ersion by majority v oting on span-le v el label Figure # Positi v e term # Ne g ati v e term Document-le v el label 2(a) 0 2 NEGA TIVE 2(b) 1 2 NEGA TIVE 2(c) 2 0 POSITIVE 2(d) 2 1 POSITIVE 2(e) 0 0 NEUTRAL 2(f) 2 2 NEUTRAL 3.2. Model b uilding After the dataset b uilding process, we split the dataset with a ratio of 80:20 (standard split), 80% of the data will be used as t raining data and 20% of the data will be used as testing data. W e further split the training data with the same ratio, 80% of data will be used as training data and 20% of data will be used as v alidation data. Indonesian J Elec Eng & Comp Sci, V ol. 38, No. 2, May 2025: 1353–1366 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1359 W e use a v ariant of IndoBER T models [24] in analyzing sentiment on a dataset that discusses natu- ral en vironment in Indonesian Instagram, namely IndoBER T -lite-base-p2, IndoBER T -lite-lar ge-p2, IndoBER T - base-p2, and IndoBER T -lar ge-p2. IndoBER T model is a pre-trained language model that w as trained on Indone- sian corpus Indo4B which is collected from Indonesian W ikipedia, ne ws articles, blogs, and social media, with a size of approximately 4 billion Indonesian w ords. The model is designed to be ne-tuned on natural language processing tasks such as SA, named entity recognition, question answering, and summarization. IndoBER T is constructed based on tw o architectures: BER T [20] and ALBER T [36]. IndoBER T -base and IndoBER T -lar ge architectures follo w B E R T B AS E (12 layers, 768 hidden sizes, 12 self-attention heads, total parameters of 110M) and B E R T LAR GE (24 layers, 1,024 hidden sizes, 16 se lf-attention heads, total parameters of 340 M) architectures respecti v ely . IndoBER T -lite-base and IndoBER T -lite-lar ge architectures follo w ALB E R T B AS E (12 layers, 768 hidden sizes, 12 self-attention heads, total parameters of 12 M) and ALB E R T LAR GE (24 lay- ers, 1,024 hidden sizes, 16 self-attention heads, total parameters of 18M) architectures respecti v ely . W illie et al. [24] used tw o phases for training IndoBER T . In the rst phase (p1), IndoBER T w as trained on the Indo4B corpus with a maximum sequence length of 128. Then, in the second phase (p2), IndoBER T w as further trained on the same corpus with a maximum sequence length of 512. The second phase scheme with a longer sequence allo ws the model to understand the meaning of w ords based on the surrounding conte xt more deeply and bet- ter . W e will use training data to ne-tune all v ariants of IndoBER T models and v alidation data to adjust the h yperparameters setup used. Then, W e use the ne -tuned IndoBER T model to classify sentiment on testing data to “POSITIVE” (ha ving positi v e sentiment), “NEGA TIVE” (ha ving ne g ati v e sentiment), or “NEUTRAL” (ha ving neutral sentiment) labels. The simpletransformers [37] library is used to implement all v ariants of the IndoBER T model. The e v aluation metric used for measuring the performance of the ned-tuned model is macro a v erage F1-score. Macro a v erage F1-score [38] is calculated by a v eraging the F1-score v alue in each class with the same weight. The metric is particularly useful in cases of imbalanced datasets, as it pro vides a balanced e v aluation of model performance across all classes. W e use Scikit-Learn [39] library to implement the F1-score e v aluation. 4. RESUL TS AND DISCUSSIONS This section discusses the results of our research including the dataset-b uilding result (subsection 4.1) and model-b uilding results (subsection 4.2). 4.1. Data b uilding r esult and discussion W e b uilt a w ord cloud to g ain insight i nto what is being discussed based on frequently occurring w ords in the dataset. F igure 3 sho ws w ord cloud per label. Figure 3(a) sho ws w ords that often appear in data labeled as “POSITIVE”. Som e e xamples of the w ords that oft en appear are “indonesia” (Indonesia), “lingkungan hidup” (en vironment), “lingkungan” (en vironment), “k e giatan” (acti vity), and “sampah” (trash). W e concluded that most data labeled as “POSITIVE” discusses en vironmental acti vities related to trash in Indonesia . Figure 3(b) sho ws w ords that often appear in data labeled as “NEUTRAL”. Some e xamples of the w ords that often appear are “sampah” (trash), “indonesia” (Indonesia), “bahan bakar” (fuel), “lingkungan” (en vironment), and “lim- bah” (w aste). W e concluded that most data labeled as “NEUTRAL” discusses w aste and fuel in the Indonesian en vironment . Figure 3(c) sho ws w ords that often appear in data labele d as “NEGA TIVE”. Some e xamples of the w ords that often appear are “indonesia” (Indonesia), “polusi udar a” (air pollution), “polusi” (pollution), “lingkungan” (en vironment), and “sampah” (trash). W e concluded that most data labeled as “NEGA TIVE” discusses air pollution and trash in the Indonesian en vironment. B ased on the w ord cloud of the three labels, the frequently occurring w ords are w aste and en vironment. These w ord clouds can then be utilized as a support system for the go v ernment or stak eholders to better manage the presence of w aste in the en vironment based on public perceptions. As mentioned in subsection 3.2, for the e xperiment, we use 20% of data for the test set. From the remaining 80% data, we use 20% of them for the v alidation set while the rest is for the training set. The dataset distrib ution of training, v alidation, and testing data can be seen in T able 2. T able 2 sho ws that the dataset is quite imbalanced where the posit i v e label becomes the majority label. This imbalanced proportion may gi v e challenges in the modeling pr o c ess. In this case, if we do not apply the proposed ltering process, we may obtain a more e xtreme imbalanced dataset. Indonesian sentiment analysis in natur al en vir onment topics (Christofer Octo vianto) Evaluation Warning : The document was created with Spire.PDF for Python.
1360 ❒ ISSN: 2502-4752 (a) (b) (c) Figure 3. W ord cloud for data per label as: (a) ”POSITIVE”, (b) ”NEUTRAL”, and (c) ”NEGA TIVE” T able 2. Distrib ution of sentiment in dataset Dataset NEGA TIVE POSITIVE NEUTRAL T otal T raining 545 800 184 1,529 V alidation 136 201 46 383 T esting 171 250 58 479 T otal 852 1,251 288 2,391 4.2. Model b uilding r esults and discussions F or the model b uilding e xperiment, we adapted the h yperparameter setup from [20] that can be seen in T able 3. W e used the NVIDIA DGX-A100 system to train and e v aluate all the models. Our e xperiment result can be seen in T able 4. The best F1-score on testing data is achie v ed by the model based on IndoBER T - lar ge-p2 with the v alue of 72.44%. F or more detail, it can be seen that the bigger the model’ s parameter the better the model performance. Therefore, it w ould be inte resting to use a model with bigger parameters for the task. Moreo v er , it w ould also be interesting for future w ork to train the IndoBER T model on an Instagram dataset and use it for the task. K oto et al. [40] sho ws that the IndoBER T model trained on the dataset with the same type as a dataset for a task can achie v e better performance compared to the IndoBER T model trained on general data lik e W ikipedia and Ne ws data. According to [40], their IndoBER T weet model (IndoBER T which w as trained on Indonesian tweets) surpassed empirically the other IndoBER T v ariants trained on general data in solving NLP tasks with tweet datasets. T o see the most dif cult label to predict by the best model, we created a confusion matrix as in Figure 4. Figure 4 sho ws the most dif cult label to predict is the “NEUTRAL” label. This phenomenon is e xpected because neutral sentiment is in the middle of polarity between ne g ati v e and positi v e sentiment. Hence, it could contain opinion terms that are slightly ne g ati v e or slightly positi v e. Moreo v er , in our Dataset training in T able 2, data with the label “NEUTRAL” is the fe west which mak es it more challenging for the model to learn this kind of data. F or future w ork, it might need to add more data with a “NEUTRAL” l abel to help impro v e model prediction. The researchers [41], [42] s ho ws that a model can perform well once the y are gi v en a suf cient quantity of data. Indonesian J Elec Eng & Comp Sci, V ol. 38, No. 2, May 2025: 1353–1366 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1361 T able 3. Hyperparameter setup to ne-tune IndoBER T model [20] Model Batch size Epoch Learning rate IndoBER T -lite-base-p2 32 15 4 × 10 − 5 IndoBER T -lite-lar ge-p2 16 15 1 × 10 − 5 IndoBER T -base-p2 32 15 4 × 10 − 5 IndoBER T -lar ge-p2 16 15 4 × 10 − 5 T able 4. Fine-tuned IndoBER T results on testing data Model F1-score IndoBER T -lite-base-p2 45.79% IndoBER T -lite-lar ge-p2 46.66% IndoBER T -base-p2 68.49% IndoBER T -lar ge-p2 72.44% Figure 4. Confusion matrix of model based on IndoBER T -lar ge-p2 F or further analysis, we conducted qualitati v e analysis on a fe w samples of incorrectly predicted l a- bels. Figure 5 sho ws sample of incorrectly predicted to ha v e “POSITIVE” label and its annotation results for annotator 1 in Fi gure 5(a), annotator 2 in Figure 5(b), and annotat or 3 in Figure 5(c). Then, Figure 6 sho ws sample of incorrectly predicted to ha v e “NEGA TIVE” label and its annotation results for annotator 1 in Figure 6(a), annotator 2 in Figure 6(b), and annotator 3 in Figure 6(c). Finally , Figure 7 sho ws sample of incorrectly predicted to ha v e “NEUTRAL” label and its annotation results for annotator 1 in F igure 7(a), annotator 2 in Figure 7(b), and annotator 3 in Figure 7(c). As can be seen in T able 1, the majority v oting code we used forces data to be represented by a document-le v el label e v en if the data contains multiple e xpression terms (span-le v el label). In each of the samples, each annotator result has multiple e xpression terms (“Exp Positi v e” and “Exp Ne g ati v e”), b ut no document-le v el label from the majority v oting can represent them well. There- fore, the o v erall e xpression terms in data may not be well represented by the nal label, so this process could mislead the model in predicting the nal label. Instead of directly using a model to predict the nal label, it w ould be interest ing to use ensemble learning [43], [44] to determine the nal label so it can accommodate multiple e xpressions terms contained in the data. W e can utilize a more granular form of SA, such as sentiment term e xtraction [45], [46] to identify all e xpression terms and their labels better as the dataset we ha v e b uilt contains such information. Also, The main focus of Instagram is to con v e y information through visual compo- nent. Therefore, it could be interesting to comprehend sent iment polarity through Instagram visual component and combine it with Instagram te xt component to g auge public sentiment to w ard a certain topics (multimodal SA) [47]–[49]. Indonesian sentiment analysis in natur al en vir onment topics (Christofer Octo vianto) Evaluation Warning : The document was created with Spire.PDF for Python.
1362 ❒ ISSN: 2502-4752 (a) (b) (c) Figure 5. Example of annotator result of data that w as incorrectly predicted to ha v e the label “POSITIVE”: (a) 1’ s, (b) 2’ s, and (c) 3’ s (a) (b) (c) Figure 6. Example of annotator result of data that w as incorrectly predicted to ha v e the label “NEGA TIVE”: (a) 1’ s, (b) 2’ s, and (c) 3’ s Indonesian J Elec Eng & Comp Sci, V ol. 38, No. 2, May 2025: 1353–1366 Evaluation Warning : The document was created with Spire.PDF for Python.