Indonesian J our nal of Electrical Engineering and Computer Science V ol. 37, No. 2, February 2025, pp. 1168 ∼ 1182 ISSN: 2502-4752, DOI: 10.11591/ijeecs.v37.i2.pp1168-1182 ❒ 1168 Recognizing geographical locations using a GAN-based text-to-image appr oach Dina M. Ibrahim, Amal A. Al-Shar gabi Department of Information T echnology , Colle ge of Computer , Qassim Uni v ersity , Buraydah, Saudi Arabia Article Inf o Article history: Recei v ed Jun 11, 2024 Re vised Sep 13, 2024 Accepted Sep 30, 2024 K eyw ords: AttnGAN model Deep learning Generati v e adv ersarial netw orks Location-based services Road infrastructure T e xt reading T e xt-to-image ABSTRA CT Generating photo-realistic images that align with the te xt descriptions is the goal of the te xt-to-image generation (T2I) model. The y can assist in visualizing the descrip- tions thanks to adv ancements in machine learning algorithm s. Using te xt as a source, generati v e adv ersarial netw orks (GANs) can generate a series of pictures that serv e as descriptions. Recent GANs ha v e allo wed oldest T2I models to achie v e remarkable g ains. Ho we v er , the y ha v e some limitations. The main tar get of this study is to address these limitations to enhance the te xt-to-image generation mode ls to enhance location services. T o produce high-quality photos utilizing a multi-step approach, we b uild an attentional generating netw ork called AttnGAN. The ne-grained image-te xt match- ing loss needed to train the AttnGAN’ s generator is computed using our multimodal similarity model. W ith an inception score of 4.81 on the P atternNet dataset, our AttnGAN model achie v es an impressi v e R-precision v alue of 70.61 percent. Because the P atternNet dataset comprises photographs, we’ v e added v erbal descriptions to each one to mak e it a te xt-based dataset instead. Man y e xperiments ha v e sho wn that At- tnGAN’ s proposed attention procedures, which are critical for te xt-to-image produc- tion in comple x circumstances, are ef fecti v e. This is an open access article under the CC BY -SA license . Corresponding A uthor: Dina M. Ibrahim Department of Information T echnology , Colle ge of Computer , Qassim Uni v ersity Buraydah, Saudi Arabia Email: d.hussein@qu.edu.sa 1. INTR ODUCTION Synthesizing images from te xt descriptions has lately been a hot topic of research because of the intro- duction of generati v e adv ersarial netw orks. It’ s a e xible and easy method for creating conditional images, with considerable impro v ements in visual realism, di v ersity , and semantic congruence in recent years. Ho we v er , the discipline still f aces se v eral obstacles that will necessitate additional research, such as permitting the creation of high-resolution photos containing man y objects and generating appropriate and trustw orth y e v aluation criteria that correlate with human judgment. Deep learning has enabled signicant breakthroughs in computer vision applications and image processing techniques in recent years, and these adv ancements ha v e had a signicant impact on the eld. F or e xample, the eld of picture synthesis, which is the act of creating ne w images and changing e xisting images, is one such e xample. In addition to its man y practical uses in areas such as art creation, picture editing, virtual reality (VR), computer -aided design, and video g ames, image synthesis is an important task to perform. In their 2014 article “Generati v e Adv ersarial Netw orks, ” Goodfello w et al. [1] detailed the design of the technique and pro vided the rst empirical proof of it. In generati v e adv ersarial netw orks (GANs), the J ournal homepage: http://ijeecs.iaescor e .com Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1169 adv ersarial learning concept trains both the generator and the discriminator . A generator model is emplo yed to produce images by taking input points from a latent space and combining them wit h a discriminator model that sorts images into tw o cate gories: real (from the dataset) and f ak e (from the gene rator) [1], [2]. Finding the possible distrib ution of actual data samples and then using that distrib ution to create ne w samples is what GAN is all about. There has been a great deal of research into GANs si nce its inception because of the v ast potential uses for them in elds such as image and vision computing, and v oice and language processing. Due to the adv ent of GANs, supervised and unsupervised training of generati v e models for images are no w feasible [1], [3]. GANs ha v e aroused a great deal of attention and ha v e resulted in signicant adv ance- ments in image synthesis research. The picture synthes is challenge w as framed as a g ame with tw o players between tw o competing articial neural netw orks, according to the researchers. Unlik e a discriminator net- w ork, a generator netw ork is trained to generate realistic samples, whereas a discriminator netw ork is trained to discern between actual and generated images [4], [5]. A discriminator netw ork is de v eloped to distinguish between genuine and produced images. The generator’ s training purpose is to decei v e the discriminator during the training process. So f ar , this approach has been successfully used a number of applications, including rep- resentation learning, style transfer , image-to-image translation, data augmentation, high-resolution human f ace synthesis, image super -resolution, and image in-painting [6], [7]. The goal of te xt-to-image (T2I) synthesis is to generate an image that accurately reects the m eaning of a te xt description. Some of the most recent studies ha v e concentrated on this topic [8], [9]. As an analogy , T2I can be vie wed as the in v erse of image captioning, in that it accepts an image as input and produces a te xt description of that picture as an output. Ho we v er , while the approaches gi v en in this study can be applied to a wide range of picture domains, the majority of T2I research is focused on methods for creating aesthetically realistic images that are photographic [10]. The goal of a T2I model is to produce images that are photo- realistic and semantically compatible with the descriptions of the te xt [11]. The y can assist in visualizing the descriptions thanks to adv ancem ents in machine learning algorithms. Using te xt as a source, GANs can generate a series of pictures that serv e as descriptions. Unsupervised machine learning includes generati v e model methods [12]. Ne w GANs ha v e allo wed current T2I models to achie v e remarkable results. Ho we v er , the y ha v e some limitations. A signicant step to w ard articial i ntelligence similar to that of humans can be achie v ed by de v eloping a system that comprehends the interplay between visual perception, language, and images, and which can generate visual representations of the meaning con v e yed by written descripti o ns . This study aims to address these limitations. The creation of GAN made unsupervised training of generati v e models for images possible. GANs ha v e attracted the interest of man y researchers and propelled the eld’ s ef forts in the area of image synthesis. This study in v estig ated the ef fects of an attentional generati v e adv ersari al netw ork (At tnGAN) to syn- thesize ne-grained te xt into images. While earlier studies ha v e e xplored the impact of a generati v e adv ersarial netw ork, the y ha v e not e xplicitly addressed its inuence on a multi-stage process, we b uild an attentional generating netw ork to enable the AttnGAN to produce high-quality photos. In order to train the AttnGAN’ s generator , we pro vide a deep attentional mul timodal similarity model that computes the ne-grained image-te xt matching loss. 2. B A CKGR OUND AND RELA TED W ORK IN T2I USING GANS A recent study proposed in [13] query that GAN is a unique scene retrie v al frame w ork that uses AttnGAN im ages as query images. According to the ndings of the e xperiments, the frame w ork that has been proposed is capable of accurately retrie ving scenes and enables users to locate the scenes that the y ha v e pick ed. The y used tw o kinds of data sets. The COCO dataset consists of daily scene images and the annotations that go along with them. There are 82,783 training images in the dataset, each of which has v e descriptions. Three mo vies were selected from the MP-II MD dataset: “BadSanta, ” As “Good As It Gets, ” and “Harry Potter and the Prisoner of Azkaban, ” which had 430, 538 and 592 scenes, respecti v ely , with an a v erage of 100 frames each scene and a total of 153,320 frames. Subjecti v e e v aluation results. The a v erage scores from 25 subjects are sho wn in these results. A score of 1 represents “not rele v ant, ” while a sc ore of 5 represents “rele v ant, ” proposed frame w ork (PF): 4.96, BL (baseline method): 4.76, comparati v e method 1 (CM1) is 3.20, comparati v e method 2 (CM2) is 4.48, comparati v e method 3 (CM3) is 4.44, and comparati v e method 4 (CM4) is 4.88. The y used Recall@k for the quantitati v e e v aluation criterion and dened frames included in the tar get scene as our ground truth. Reco gnizing g eo gr aphical locations using a GAN-based te xt-to-ima g e appr oac h (Dina M. Ibr ahim) Evaluation Warning : The document was created with Spire.PDF for Python.
1170 ❒ ISSN: 2502-4752 Recall@k w as obtained for the inte grated dataset, which included v e mo vies: “Bad Santa”, “ As Good as it Gets”, “Hallo ween”, “Rendezv ous mit Joe Black”, and “Harry Potter and the Prisoner of Azkaban”. In the 60,000 rank, recall@k=.7. Additionally , by demonstrating the ef fecti v eness of the suggested frame w ork, it is possible to conrm the usefulness of the generated images, which are not aesthetically pleasing. Zhang et al. [14], both rendering graphics from te xt descriptions and stylizing them based on a gi v en style image were the goals of the authors. GAN systems can generate a stylized picture directly from an image and te xt descripti on, rather than generating the image rst and then applying style without ha ving an y prior kno wledge of the te xt description. When the task w as brok en do wn into tw o halv es, StackGAN w as able to generate more realistic, higher -resolution images. In stage I GAN, lo w-quality visuals are generated based on the te xt to approximate the meaning (lik e the methods described abo v e). In stage II GAN, the te xt and output from stage I are used to b uild a higher -resolution rendition with additional details. Data pairs in the type of te xt description, stylized image are required for their project’ s requirements. The “image captioning” data sets, which comprise data pairings in the form “te xt description, image, ” were not a v ailable, thus we used neural style transfer to generate styled images for these datasets. Man y image captioning databases are a v ailable, such as these: COCO has more than 200,000 images co v ering a wide range of subjects, each with v e accompan ying captions. CUB-200-2011: 6,033 images of birds from 200 dif ferent species. T o train COCO, we needed a lar ge number of photographs, and in our instance, we needed to generate stylized v ersions of all of these photos, possibly se v eral times, in order to handle each of COCO’ s man y styles. This is wh y we opted to not only use a smaller dataset, b ut also limit the number of species in the dataset, so that we could reduce training and data production tim e. In order to v erify that each group had enough samples, we separated the data by species, not as a random sampling. F or the rst 451 photos, we created stylized images in tw o styles, which related to eight bird species (lar ger than a v erage cate gories for the CUB dataset). Because it tak es too long to collect enough stylised training data and e x ecute a h yperparameter search with enough iterations each time to determine the ideal settings, no rele v ant quantitati v e results were obtained. Similarly , attention-dri v en multi-stage renement for ne-grained te xt-to-image generation is pro- posed by the authors in their w ork on the AttnGAN netw ork, in [15]. By focusing on the important w ords in the te xtual description, the AttnGAN, a ne w attentional generati v e netw ork, is able to generate ne-grained features at v arious picture sub-re gions. This paper has used AttGAN for COCO dataset: train 8,855 images with 10 caption/image and CUB dataset: train 80K images with 5 caption/images. The result of the AttnGAN 4.36 ± .03 inception score and 67.82 ± 4.43 R-precision for CUB dataset and 25.89 ± .47 inception score and 85.47 ± 3.69 R-precision for COCO dataset. By impro ving the best reported inception score by 14.14% on the CUB dataset and 170.25% on the more dif cult COCO dataset, our AttnGAN f ar surpasses state-of-the-art GAN models. P articularly important for complicated scene te xt-to-image generation, e xtensi v e e xperimental data sho w that the AttnGAN’ s postulated attention processes w ork. An adv ersarial netw ork (A CGAN) is designed to produce high-quali ty images with a resolution of 1024x1024 in [16]. The initial proposal is a multi-tiered cascade architecture for te xt-to-image synthesis. In order to train the model to produce images with photorealistic detail, we progressi v ely add more layers, with each layer using the outputs and w ord v ectors of the preceding layer as inputs. Second, to train the generation, we insert a deep attenti on a l multimodal similarity model into the netw ork and match w ord v ectors and images in a common semantic space to calculate a ne-grained m atching loss. Results sho w that the suggested model outperforms AttenGAN on the CUB dataset with an inception score of 4.48 and the Oxford-102 dataset with an inception score of 6.42 percent, respecti v ely . When it comes to te xt-generated images, the A CGAN model produces better results and more accurate results. The y can assist in visualizing the descriptions thanks to adv ancem ents in ma chine learning al gorithms. In order t o generate a series of visual representations of te xtual descriptions, GANs can be emplo yed [17]. A subset of unsupervised machine learning is generati v e model algorithms. The v ersatility and ease of use of GANs mak e them ideal for a wide range of tasks, including model creation and visualization based on user input. The y ha v e applications in house design and architectural planning. The y can also be utilized to create animations, b ut the y can speed up the process, making them ideal for use in virtual g ames. The adv ertising and clothing industries are tw o more that can benet from GANs. Comparing the suggested model’ s output to that of mirror GANs, the results are mi x ed. The proposed model appears to produce unique images that closely resemble birds, as indicated by the inception score of 5.089. The proposed model appears to produce uni que images that closely resemble birds, as indicated by the Indonesian J Elec Eng & Comp Sci, V ol. 37, No. 2, February 2025: 1168–1182 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1171 inception score of 5.089. The generated images’ v ariability increases as the inception score rises. In a similar v ein, a high inception score indicates that the Google inception model made a precise classication. Ev ery machine learning problem core of e v ery dataset. The three sets of e xtensi v ely utilized dataset s in T2I research include Oxford-120 Flo wers [18], CUB-200 Birds [19], and COCO [20]. W ith each dataset consisting of approximately 10,000 photos, Oxford-102 Flo wers [18] and CUB-200 Birds [19] are on the smaller side. Each image depicts a single object, and each image has 10 single sentence captions that ha v e been collected using Amazon Mechanical T urk (AMT) w ork ers in the US [21]. All of the emplo yees were instructed to describe themselv es in detail. In contrast, COCO [20] consists of approximately 123,000 photos, each of which has v e captions that were generated by humans and collected via AMT . Ev ery emplo yee w as gi v en specic i nstructions to describe the scene in detail while a v oiding certain terms and phrases. These included not using the w ord ”there” in the rst line, not discussi ng an ything that could happen in the past or future, not using formal names, and using a minimum of 8 w ords. The of cial 2014 COCO split is used in most of T2I w orks. Chen et al. [22] pro vide more details on the captions collection process. Unlik e the Oxford-102 Flo wers and CUB-200 Birds datasets, COCO images usually contain man y , often interacting objects. T able 1 sho ws a list of the most common datasets used for T2I illustrati ng the total number of training, testing, captions, and object cate gories per each dataset. T able 1. A list of the most common datasets for T2I approach Dataset T rained i mages T ested images T otal images Captions of image Object cate gory Oxford-102 Flo wers 7,034 1,155 8,189 10 102 CUB-200 Birds 8,855 2,933 11,788 10 200 COCO 82,783 40,504 123,287 5 80 F or our proposed w ork model, input noisy images with suitable captions, generator netw ork, discriminator netw ork, and training methods are essential parts that should be focused on. In this study , our research method consists of six steps which are illustrated in Figure 1. Figure 1. Our proposed research method − Step 1: input a randomly generated noise signal and te xt to the generator . The generator will generate random images that do not ha v e an y rele v ance to the real w orld. − Step 2: no w , we are going to start trai ning the discriminator . W e input the random images generate d by the generator with a sample of the real images of the dataset. − Step 3: the discriminator gi v es by then the probabilities of all the inputted images being real. − Step 4: compare with the actual labels (1 for real images and 0 for f ak e images). Calcul ate the loss error and back-propag ate it to update the weights of the discriminator . − Step 5: run the generated images through the discriminator , b ut this time without the real images. The discriminator will most probably output probabilities of the images being f ak e. Reco gnizing g eo gr aphical locations using a GAN-based te xt-to-ima g e appr oac h (Dina M. Ibr ahim) Evaluation Warning : The document was created with Spire.PDF for Python.
1172 ❒ ISSN: 2502-4752 − Step 6: we no w compare the probabilities in the output with the e xpected label not the actual. Then, calculate the loss error and backpropag ate it to update the weights of the generator . Figure 2 sho ws samples of our e xpected proposed model. After training, the generator is able to generate realistic images that are independent from the real images (not e xact, b ut close), and the discriminator thinks the y’ re correct! Figure 2. Examples of our predicted output images The performance of the T2I models based on GAN techniques is e v aluated based on se v eral met rics: inception score (IS), Fr ´ echet inception distance (FID), R-precision, and nall y human rank. Score for IS: generati v e adv ersarial netw ork models produce synthetic images, and the IS [23] is a measure that objecti v ely assesses their quality . T w o characteristics of a set of produced images—thei r di v ersity and their quality—are aimed at by the score. The FID [24] measures ho w f ar apart the distrib utions of real and f ak e images are for features that a netw ork that has already under gone training pulls out. When comparing GANs, the FID is more reliable than the IS and can detect a wider v ariety of disruptions. The visual-semantic similarity between te xt descriptions and generated im ages is measured by R-precision [15], which ranks retrie v al results based on e xtracted picture and te xt attrib utes. Human rank: for qualitati v e e v aluation, human rank is used. In the CUB and Oxford test sets, 50 te xt descriptions were chosen at random, and the created model generated 5 images for each sentence [16]. The v e images and accompan ying sentences are e xplained to se v eral people, who then rank the image quality in v arious w ays before calculating an a v erage ranking to assess the qual ity and di v ersity of t h e created images. T able 2 presents a summary of the GAN models’ performance metrics. T able 2. A summary of the GAN models performance metrics Model FID IS R-precision Human rank Dataset GAN-INT -CLS 68.79 2.88+-.04 - - C UB 60.62 7.88+-.07 - - CO CO GA WWN - 3.62+-.07 - - CUB StackGAN 51.89 3.70+-.40 - - C UB 74.05 8.45+-.03 - - CO CO StackGAN++ 15.30 4.04+-05 - - C UB 81.59 8.30+-.10 - - C OCO HDGAN - 4.15+-.05 - - CUB - 11.86+-.18 - - COCO AttnGAN 25.72 4.36+-.03 67.82+-4.43 - CU B 35.49 25.89+-.47 85.47+-3.69 - COC O MirrorGAN 26.80 4.56+-.05 86.0+-.55 - CU B 37.86 26.47+-.41 86.0+-1.05 - COC O DM-GAN 16.09 4.75+-.07 72.31+-.91 - C UB 32.64 30.49+-.57 88.56+-.28 - CO CO Obj-GAN - - - - CUB 21.21 32.79+-.21 93.39+-2.08 - CO CO GAN-INT -CLS - 2.88 ± .04 - 2.81 ± .03 CUB - 2.66 ± .03 - 1.87 ± .03 Oxford-102 - 7.88 ± .07 - 1.89 ± .04 ms COCO GA WWN - 3.62 ± .07 - 1.99 ± .04 CUB Indonesian J Elec Eng & Comp Sci, V ol. 37, No. 2, February 2025: 1168–1182 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1173 3. MA TERIALS AND METHODS 3.1. Datasets f or the study In this w ork, W e will apply T2I-AttnGAN to the satellite data set, to get the needed datasets for t his study we used tw o well-kno wn dif ferent scene datasets, the P atternNet dataset, and the Merced dataset and we proposed this dataset to enhance map services. 3.1.1. The P atter nNet dataset Image retrie v al using remote sensing w as the purpose of collecting the lar ge-scale, high-res olution remote sensing dataset kno wn as P atternNet [25]. There are 38 dif ferent cate gories, and each one has 800 images that are 256x256 pix els in size. Images used by P atternNet for se v eral US cities were sourced from Google Earth via the Google Maps API. Figure 3 sho ws some e xamples of each class’ s images. Figure 3. Sample of the P atternNet dataset 3.1.2. The Mer ced dataset The UC-Merced land use dataset, which is f amous for its 2100 high-resolution remote sensing photos or g anized into 21 classes is the rst [26]. Figure 4 sho ws some e xamples of each class’ s images. This dataset is dif cult to classify since se v eral classications, such as ”dense, ” ”medium, ” and ”sparse” residential, ha v e a lot of o v erlap. This dataset has been used e xtensi v ely to test techniques for classifying aerial scenes. Figure 4. Sample of the Merced dataset The datasets used in this study are the satellite datasets to enhance map services. During the annota- tion, a fe w guidelines that needed to be adhered to when the sentences were being generated were considered: − P ay attention to the most prominent items; little ones might not be w orth your time. Reco gnizing g eo gr aphical locations using a GAN-based te xt-to-ima g e appr oac h (Dina M. Ibr ahim) Evaluation Warning : The document was created with Spire.PDF for Python.
1174 ❒ ISSN: 2502-4752 − Refrain from dra wi n g attention to things’ hues (lik e blue cars) and instead focus on their density and pres- ence. T o dif ferentiate it from an y generic parking lot (do wnto wn, for e xample), it is necessary to add the w ord ”airport” when referring to a parking lot located in an airport. − Stay a w ay from conjunctions and punctuation. 3.2. Our AttnGAN-based T2I pr oposed model Figure 5 sho ws the name of the GAN model emplo yed in the suggested system, which is AttnGAN. The AttnGAN and the deep attentional multimodal similarity model (D AMSM) are the tw o main parts of this model. The planned AttnGAN’ s architecture is sho wn in the Figure 5. The D AMSM supplies the generati v e netw ork with ne-grained picture-te xt matching loss, while each attent ion model autonomously collects the criteria (i.e., the most rele v ant w ord v ectors) for creating distinct image sub-re gions. Figure 5. The architecture of the proposed AttnGAN model 3.3. Attentional generati v e adv ersarial netw orks GAN form the basis of the most current te xt-to-image synthesis algorithms. The criterion for GAN- based picture generation is often the complet e te xt description encoded into a global phrase v ector . Despite the remarkable outcomes, producing high-quality images is hindered by conditioning GAN solely on the global sentence v ector , which does not pro vide crucial w ord-le v el ne-grained information [15]. An adv er - sarial attention-generating netw ork to generate images from te xt with a ner le v el of detail, AttnGAN enables attention-dri v en multi-stage rening. There are tw o ne w parts to the model. As a rst step, we b uild an atten- tional generati v e netw ork, which uses a pre-e xisting attention mechanism to selecti v ely generate visual re gions by highlighting w ords with high conte xtual rele v ance. More precisely , e v ery w ord in the sentence is encoded into its o wn w ord v ector in addition to the global sentence v ector that contains the natural language description. Starting with a lo w-resolution image, the generati v e netw ork uses the global sentence v ector . Afterw ards, it forms a w ord-conte xt v ector by querying w ord v ectors using an attention layer with the picture v ector in each sub-re gion [15]. The model then uses a multi-modal conte xt v ector that is formed by com- bining the re gional image v ector with the matchi ng w ord-conte xt v ector to produce additional image features in the nearby sub-re gions. A higher -resolution image with more information at each le v el is ef ciently pro- duced by this. D AMSMs are the other parts of the AttnGAN. An attention m ethod allo ws the D AMSM to use both coarse-grained w ord-le v el information and global sentence-le v el information to calculate the sentence- to-generated-image similarity . T o train the generator , the D AMSM adds a ne-grained image-te xt matching loss. Attentional Model: In order for the attentional generati v e netw ork to produce realistic visuals with both sentence-le v el and w ord-le v el requirements, the nal goal function is dened as: L = L G + λL D AM S M , w hr er L G = m − 1 X i =0 L G i (1) Here, the h yperparameter λ is used to mak e sure that the tw o parts of the equation are balanced. The rst term is the GAN loss that jointly approximates conditional and unconditional distrib utions [15]. Generator model: the AttnGAN’ s generator Gi is paired with its discriminator Di during the i-th step. Model for Discrim- inator: In contrast to training Gi, each discriminator Di is taught to minimize the cross-entrop y loss in order to identify inputs as either real or f ak e. Indonesian J Elec Eng & Comp Sci, V ol. 37, No. 2, February 2025: 1168–1182 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1175 3.4. Deep attentional multimodal similarity model Using a common semantic space, the D AMSM learns to associate image and te xt elements with one another using tw o neural netw orks. As a result, it is able to generate images with a ner grain of loss by determining the de gree of similarity between the image and te xt at the w ord le v el. By utilizing bidirectional long short-term memory (LSTM), the te xt encoder is able to e xtract semantic v ectors from the te xt descrip- tion. T w o hidden states, one for each direction, are associated with each w ord in the bidirectional LSTM. Afterw ards, we b uild an attention model that combines the tw o hidden states (query) of each w ord to produce a re gion-conte xt v ector [15]. In order to con v ert pictures into meaningful v ectors, the picture encoder uses a con v olutional neural netw ork (CNN). V arious sub-re gions of the image are learned by the CNN’ s intermediate layers, while the image’ s global features are learned by the subsequent layers. W e present the attentional generati v e netw ork (AttnGAN) and the D AMSM, tw o ne w attention m o de ls with distinct functions; i) the AttnGAN is able to autonomously generate v arious pi cture sub-re gions by using the attention mechanism in the generati v e netw ork to choose w ord-le v el conditions and ii) applying an atten- tion mechanism enables the D AMSM to calculate the LD AMSM, a loss function for ne-grained te xt-image matching. Note that LD AMSM is only applied to the Gm-1 output since the nal purpose of the AttnGAN is to produce big images from that generator . W e made an ef fort to use LD AMSM on all resolution photos produced by (G0, G1,..., Gm-1). The computational cost went up, ne v ertheless, and performance stayed the same. 3.5. Our dataset P atter nNet Se v eral guidelines for the construction of the sentences were considered during the annotating proces s: the rst rule of focus is to ignore minor details in f a v or of the major , dominating items. Second, focus on describing what is actually there in the situation rather than what is not. Thirdly , instead of xating on the quantity of objects, try using more general terms lik e “man y”, “fe w”, and “man y”. F ourth, focus on the presence and density of objects rather than their color (lik e blue cars). Fifth, the w ord ”airport” must be included when referring to a parking lot within an airport in order to dif ferentiate it from an y other type of parking lot, such as one located do wnto wn. Then sixth, a v oid using conjunctions or punctuation. In Figures 6 and 7, readers will observ e a fe w results from our dataset. In our study , we ha v e only used images from P atternNet datasets from [25] and we added a te xtual description for each image to transform it into a te xt-based dataset. There are a total of 38 classes in P atternNet, with 800 images of 256 × 256 pix els each. The total number of photos we used w as 4,800, as sho wn in T able 3, because we pick ed 24 classes and 200 images from each class of these datasets. W e created tw o folders (P atternNet) containing 24 classes for the images used and another folder (P atternNet.te xt) containing 24 classes for the description in which we annotated the images. In the 200 images we chose from 24 dif ferent classes we annotated each image in a separate te xt le with three dif ferent sentences; 14,400 sentences were handed do wn as a result. Figure 6. Example for images with three sentences for each image Reco gnizing g eo gr aphical locations using a GAN-based te xt-to-ima g e appr oac h (Dina M. Ibr ahim) Evaluation Warning : The document was created with Spire.PDF for Python.
1176 ❒ ISSN: 2502-4752 Figure 7. Another e xample for images with three sentences for each image T able 3. Dataset classes in our study 38 original classes 24 classes we chose Beach Airplane Chaparral Baseball eld costal mansion Bask etball court forest Closed road cemetery Crossw alk Golfe course Dense residential Oil well Ferry terminal Bridge F ootball eld P arking space Harbor Free w ay Intersection Runw ay marking Mobile home park tennis court Nursing home stor ge tank Oil g as eld F ootball eld Ov erpass Ferry terminal P arking lot Dense residential Rail w ay Crossw alk Ri v er Closed road Runw ay Bask etball court Shipping yard Baseball eld Solar panel Airplane Sparse residential Harbor Swimming pool Intersection T ransformer station Mobile home park W aste w ater treatment plant Nursing home - Oil g as eld - Ov erpass - P arking lot - Ri v er - Runw ay - Shipping yard - Solar panel - Sparse residential - Swimming pool - T ransformer station - W aste w ater treatment plant - Indonesian J Elec Eng & Comp Sci, V ol. 37, No. 2, February 2025: 1168–1182 Evaluation Warning : The document was created with Spire.PDF for Python.
Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 ❒ 1177 3.6. The experiment of our pr oposed model T o assess the planned AttnGAN, a lar ge amount of e xperimentation is conducted. W e be gin by delving into the AttnGAN’ s foundational elements, which encompass the attentional generating netw ork and the D AMSM. Afterw ards, we test our AttnGAN on the PetternNet dataset, which con v erts te xt to images. Our dataset contains 24 classes each class ha v e 200 images, so the total number of images are 4,800 images. Each image has its o wn description in a separate te xt le with three sentences. W e di vided the dataset into 80% for training and 20% for testing. This di vision is e x ec u t ed in each class from the 24 classes where from each class 160 images with their te xt used for t raining and 40 images with their te xt used for testing. So, the total number of the training images is 3,840 images with their te xt and the total number of the testing images is 960 images with their te xt. T able 4 lists the statistics of datasets. T able 4. P atternNet dataset statistics Dataset T raining T esting F or each class from the 24 classes 160 (image + te xt) 40 (image + te xt) T otal 3840 (image + te xt) 960 (image + te xt) Here are some e xamples of the outcomes obtained by modifying the te xt descriptions using our AttnGAN model that w as trained on P atternNet. W e put e xamples from four classes: airplane class, dense residential, harbor class, swimming pool. Figure 8 sho ws e xamples of our AttnGAN model trained on P atternNet from the airplane class using the sentences: − Airplane on runw ay . − White plane. − Red wings. Figure 8. Results obtained by modifying the te xt descriptions of the airplane class using our AttnGAN model trained on P atternNet as an e xample As in Figure 9, e xamples of our AttnGAN model trained on P atternNet from the dense residential class are represented using the sentences: − Buildings same in shape. − W ater pools appearing beside the b uildings. − T rees among the b uildings. Reco gnizing g eo gr aphical locations using a GAN-based te xt-to-ima g e appr oac h (Dina M. Ibr ahim) Evaluation Warning : The document was created with Spire.PDF for Python.