3.1. Nature of data

The multi-modality and variety of data flowing in the system and the necessity of homogenization and fusion mandated the adoption of a semantic knowledge graph to address the requirements of the project. The XR4DRAMA framework is not responsible for archiving and storing raw data files, since there is an underlying data storage facility for that purpose. Instead, the XR4DRAMA framework hosts metadata of raw data, analyses results and miscellaneous information to be mapped and fused into the knowledge base. The XR4DRAMA framework is accompanied with novel ontological models to achieve proper semantic annotation of the raw data.

The main categories of data needed to be captured in the XR4DRAMA framework were: physiological analysis, visual analysis, and textual analysis results and general information about virtual reality experiments.

The physiological signals were collected, in a controlled experiment oriented to the requirements of the project, which would detect the stress level of the subjects. The ultimate goal was to obtain and assess stress indicators directly from the skin of the subjects while experiencing different setups of the space. The analysis for these data was based on a rational [0, 1] space where values close to 0 expressed neutrality while values near 1 suggest high stress levels (Gollob et al., Reference Gollob, Kyrou, Petrantonakis and Kompatsiaris2022).

The visual analysis component consists of several machine learning (ML) models either trained from scratch or fine-tuned from other efforts. The model performing semantic segmentation (Qiu et al., Reference Qiu, Li, Liu and Huang2021) on images was trained to extract semantic labels and percentages per pixel on images while the Verge classifier (Andreadis et al., Reference Andreadis, Moumtzidou, Apostolidis, Gkountakos, Galanopoulos, Michail, Gialampoukidis, Vrochidis, Mezaris and Kompatsiaris2020) was deployed to classify the images (or video frames) to one or more classes based on context.

The analysis conducted on text was extensive, however, only few resulting assets were selected to be included in the XR4DRAMA framework due to the decision not to deep copy structures from a SOLR instanceFootnote 7 which do not serve any requirements. Such assets are: the text itself, the sentences which are contained in the text, and the named entity relations found within the text.

3.2. XR4DRAMA ontology scheme

In this subsection, we give a high-level overview of the KG schema’s structure (i.e., the XR4DRAMA ontology) and the guiding principles of each class. You can find the KG and the software developed to populate it here footnote 3. A high-level overview of the main XR4DRAMA ontology classes is shown in Figure 3.

Figure 3. High level illustration of the XR4DRAMA’s framework KG.

The initial scheme of the ontology was based on the user requirements, and the competency questions (CQs) that the domain experts defined. So, a first version of the scheme was developed only from user interaction and knowledge from domain experts. Subsequently, we also took into consideration the information that was mapped inside the KG, meaning that after the structure and the content of the messages from the various component was defined, we restructured the scheme of the ontology in order to cover the extra information that needs to be represented.

• InformationOfInterest: The fundamental entities of interest to aid in decision-making.

• Location: The location of an event is represented by this class. It may be displayed with the location’s name or with its coordinates (e.g., Vicenza).

• Metadata: All the supplementary data that is provided with the analysis results and can be used in the decision-making process is referred to as metadata.

• MultimediaObject: Indicates the type of the transmission that is used to transfer information; it can be either Audio, Textual, or Video.

• Observation: This class, which is utilized by each individual component, describes the method of assessing the date.

• Result: This class represents information about the outcome of an observation.

• Procedure: This class represents information about the procedures that should be taken during an observation (i.e., tasks that should be performed).

• Project: A project is a set of observations.

• RiskReport: This class describes the total outcome of all risk levels derived from various components. We added this class as some locations can be risky to access due to various reasons.

• User: Users are journalists, first responders and citizens, and each journalist/first responder/citizen is given a unique ID.

We also analyze the purpose of the various object type properties, that is, properties that connect instances from one class with instances from another class.

• hasMultimedia: This property identifies if the observation was provided via a textual, video, or image post.

• hasResult: This property shows how the event in the observation turned out.

• usedProcedure: This property identifies the method used to extract the information from the observation, such as whether the information was taken from the visual or textual analysis component.

• hasInformationOfInterest: This property identifies the most important data in an observation, such as the type of scene, for example airfield, and the recognized objects, such as building, automobile, etc. Domain experts suggested what was deemed important information.

• hasMetadata: This property indicates the metadata of the observation, based on the type of the observation, namely weather the observation is a result of the: (a) visual analysis component (see Table 1), or (b) textual analysis component (see Table 3).

• hasLocation: This property shows where the important information in an observation is located. Notice that the location is specified using latitude and longitude.

• consistsOf: This property lists the observations that make up a project. A project is a collection of observations with neighboring coordinates.

• hasProjectLocation: This property indicates the location of the project.

• includes: This property indicates the ID of the user that created the project.

• hasUserLocation: This property indicates the location of the user.

• hasRiskReport: This property indicates the risk level of the project, that is, if it is an emergency or not.

The subgraphs where the visual, textual, and stress level analysis messages are mapped in the KG of the XR4DRAMA framework, can be found in Figures 4, 5, and 6, respectively.

Figure 4. Subgraph of KG where the Visual Analysis message is mapped.

Figure 5. Subgraph of KG where the Textual Analysis message is mapped.

Figure 6. Subgraph of KG where the Stress Analysis message is mapped.

3.3. Points of interest management mechanism

The idea behind POIs is to create some points in an area (i.e., pins on a map) that contain crucial geospatial information, that could ease the work of first responders, and help citizens to avoid an emergency. POIs can be created or updated by any user through a phone application (which is not currently public). The user can either send an image/video which will then be analyzed by the visual analysis component, and some of the crucial information in the image/video will be then passed to a new or already existing POI, or a textual message; the pipeline for a textual message is similar. Notice that in Figure 1, we also show that the POIs receive information from the KG. Here we analyze only the information from the messages, as the information from the KG refers to some IDs that relate to the projects and the observations (see Section 3.2).

The creation of a POI is easily understandable, as if no POIs exist in the area a new one needs to be created, if an emergency occurred. On the other hand, the update of POIs is performed when other POI(s) already exist in the area, and some information in them needs to be updated, as the state of the emergency has changed. For instance, a flood affected more buildings. Below we analyze the information from a visual message that is passed to a POI when is created (see Table 1) or updated (see Table 2).

One can notice that when a POI is created the information passed from the visual analysis messages are: (i) how many people are in danger, (ii) how many vehicles are in danger, (iii) how many animals are in danger, (iv) what infrastructure was affected, (v) what objects are affected, (vi) the type of emergency, and if the emergency is a flood (vii) if a river has overflown. The aforementioned data can be dynamic, meaning that even if some are missing, the POI will still be created. Moreover, the current_user is the name of the user who sends the message, the category and subcategory characterize the area which was affected; in our running example a Education area, and more specifically, a University was affected. Finally, the coordinates are also provided in the format: [longitude, latitude]. The necessary data is the current_user, the category, the subcategory, and the coordinates. On the other hand, if a POI already exists only some information can be updated. The data which can be updated are: (i–vii).

We also analyze the information from a textual message that is passed to a POI when is created (see Table 3) or updated (see Table 4).

Table 1. Information passed from a visual message to a POI when created

Table 2. Information passed from a visual message to a POI when updated

Similarly when a POI is created the information passed from the textual analysis messages are: (i) which are the affected objects, (ii) an auxiliary label that characterizes the location, and (iii) the source text of the textual message. The aforementioned data can be dynamic, meaning that even if some are missing the POI will still be created. The necessary data is the current_user, the category, the subcategory, and the coordinates. On the other hand, if a POI already exists only some information can be updated. The data which can be updated are: (i-iii).

Table 3. Information passed from a textual message to a POI when created

Table 4. Information passed from a textual message to a POI when updated

3.4. Decision support system

One of the main goals of the DSS is to generate the severity level (SL) of a POI by fusing the information that is available to the KG, either via the Textual or the Visual Analysis outcomes. The severity score is then used from the KG to create or update the situational picture of a danger zone containing that POI. A ML approach was used in order to assess the current severity of a situation, with slight differences depending on the type of message that the DSS receives from the KB, namely Visual (2), Textual (3), simultaneously or independently.

For the Visual Analysis messages, a ML approach was developed in order to assess the weights of every category that was detected from the Visual module in an intelligent way. Using the results produced by the analysis of example frames of videos with flood scenarios by the Visual Analysis module we created an annotated dataset with 96 entries. The dataset was provided by the Alto Adriatico Water Authority (AAWA)Footnote 8, which provided us with water level measurements, from various locations along the length of the Bacchiglione river in Vicenza. That dataset was then used to train and test four well-known ML algorithms, namely the Ridge Regression Classifier (RRC), the Support Vector Machine (SVM), the Random Forest (RF) Classifier and the Decision Tree (DT) Classifier looking for the best possible accuracy score. Then, the best performing model was integrated into the DSS framework and by its interaction with the KG, the updated values of the SL of a POI can be inferred (Figure 7), see also the Accuracy Score of each model in Section 4.3. When a Visual Analysis message is generated, the KG sends the vital information the DSS needs to assess the SL of the event in real time. An example message can be seen below.

Figure 7. Machine Learning module development and KG interactions.

Using the information contained in that message, DSS is able to use the pretrained ML model in order to predict the SL of the current situation. Then, DSS generates a message for the KG with the total SL (‘Low’, ‘Medium’, ‘High’, ‘Very High’), the geolocation and the id of the current POI, as can be seen by the following example.

For the Textual Analysis messages, a similar approach was followed. Specifically, a ML method that takes into account the specific type of data sent by the KG to the DSS module has been developed as well. When a textual message is generated, KG communicates that information to DSS in real time. The following message is an example of such a textual message.

In order to train a ML model able to predict the SL of an ongoing situation given the above information, a dataset was created with random possible flood or no flood scenarios. In those scenarios, random affected objects representing real-life crisis situations were present, along with the number of each type. In order to train the ML algorithm each affected object was categorized into five greater classes: People, Animals, Vehicles, Objects, and Infrastructure (Table 5). These are the classes that are used during the prediction phase of the module, too. The event that is categorized as a “flood” or not, or the previous “severity” level of the current POI (if existing) are parameters taken into consideration in the final assessment of the situational picture too. The same four ML algorithms as in the Visual Analysis DSS module (LR, SVM, RF, DT) were used for the training procedure. The model with the best accuracy score was then integrated into the DSS framework, in order to be able to interact and update the values of the SL of the Knowledge Base. Figure 7 represents the same procedure followed in the development of the Textual Analysis DSS sub-module, too, just using a different annotated dataset during the training process.

Table 5. General Classes for affected objects

When a Textual Analysis message is generated, the available data are communicated by the KG to the DSS module. Then DSS, after classifying each affected object and taking account of the whole situational picture, generates the predicted SL of the current situation. A message is then sent to the KG using the same format and the same SL classification categories as in the visual analysis.

3.5. Danger zone creation or update

In this section we present how the danger zones are updated or created based on the severity score received from the DSS. There are three different scenarios, which we analyze in detail, that the XR4DRAMA framework considers when it receives a severity score from the DSS. The message received from the DSS, in all scenarios, has the following form.

The label severityDSS can take four discrete values which indicate the SL level of a location (‘Low’, ‘Medium’, ‘High’, ‘Very High’). The latitude and longitude indicate the location that the severity score refers to, and the project_id is a unique identifier for the broader area that the location in the messages belongs into. In the first scenario no danger zones exist in the area surrounding the point (i.e., latitude-longitude) that is contained in the message. In this case a new danger zone will be created that will be assigned the severity which is contained in the message. The danger zone is a bounding box which is created based on the latitude and longitude that exist in the DSS message, based on the following function: (latitude - 0.001 longitude - 0.001 latitude + 0.001 longitude + 0.001).

In the second scenario, a danger zone already exists in the area. If the severity score is ‘Low’ then we do not make any changes in the old danger zone.

In the third scenario, a danger zone already exists in the area. If the severity score is ‘Medium’, ‘High’, or ‘Very High’ then we apply some changes to the old danger zone following the next rules. If the new danger zone is totally contained in the old, then the area remains as it is, but if the new danger zone is not totally contained in the old danger zone, we will merge the area of the two danger zones (see Figure 8).

Figure 8. Update of Danger Zones.

3.6. Task creation mechanism

The task creation list is another function which is a result of the communication between the XR4DRAMA framework and the DSS. In this case, the DSS will recommend to the XR4DRAMA framework to attach a set of tasks to POIs, that need to be performed by the first responders for the disaster to be tackled. The use case for this service was based on a flooding scenario, in which external sensors would measure the water level of the river Bacchiglione which passes through Vicenza, and if the water level would overcome a threshold, then the DSS would recommend various tasks that need to be performed on different locations. The location of the sensor as well as the thresholds were placed and defined by the civil protection authorities of the area. Therefore, the task would be attached to POIs for the first responders to see and perform them, to have the best possible outcome in a disaster management scenario.

We provide a message as it comes from the DSS, which the XR4DRAMA framework must map in the various POIs existing on the map. Notice that where the task will be mapped is subject to the coordinates it contains.

The most crucial information in the previous message is contained in the labels: (i) name which is the title of the task that needs to be performed, (ii) the responsibility which indicates who oversees performing the task, (iii) the criticality which shows how important is the task to be performed, and (iv) the action which contains a detailed description of the task. Apart from the important information, the water level, the coordinates, and the threshold for the water level are also contained in the message.

3.7. General information-emergency contacts-legal documents

This section refers to the crucial information that accompanies a project. A project is a set of observations, where each observation corresponds to a visual, textual or stress level analysis message. More intuitively, a project is a set of observations which are clustered based on locality. For instance, a project may refer to all observations for the urban area of Vicenza. The XR4DRAMA framework is capable of attaching important information that are related to a project, such as the emergency numbers of the country and the area, the general description of the area, and the legal documents concerning drone operation and recording in public places. The general information refers to a general description of the area that the project refers to. In more detail, each time a new project is created the KG will extract from Wikipedia initially, and if it does not find any information from WikipediaFootnote 9 it will search in DBpedia (Auer et al., Reference Auer, Bizer, Kobilarov, Lehmann, Cyganiak and Ives2007), the abstract that refers to the location of the project.

The emergency data are the emergency numbers that can be used for each project, the emergency numbers are subject to the country, and sometimes region of the country. These numbers are the numbers from which one calls the police, ambulances, and/or firefighters.

The legal documents refer to the legislation concerning recording in public places for each country (and region of country in some cases), and legislation about operating drones in public spaces.

A detailed message of what the aforementioned information looks like to the user is given below, where we extract all of the previous information for the capital of Germany, Berlin.

4.1. Knowledge graph evaluation

The completeness of the KG was evaluated through a set of CQs assembled during the formation of the official ontology requirements specification document (ORSD) (Suárez-Figueroa et al., Reference Suárez-Figueroa, Gómez-Pérez and Villazón-Terrazas2009). For this reason, before constructing the KG, we asked from users to define a set of questions that they would like to be answered by the KG of the XR4DRAMA framework. The users were authority workers from Autorita’ di bacino distrettuale delle alpi orientali Footnote 10 and journalists from Deutsche Welle Footnote 11. In total a number of 32 CQs was collected, and in Figure 9 we present a sample of 10 CQs, the complete list of CQs can be found hereFootnote 3.

Figure 9. Sample of competency questions.

The completeness of the KG was found adequate, as each CQ when translated into a SPARQL counterpart returned the desired information. For instance, the second CQ from Figure 9 when translated into a SPARQL query (see Example 1), for the observation VisualMetadata_2c60537511c240c9add7fb2eb4e7459e_0 returned flood. Notice the name of the observation is the result of the text VisualMetadata_, if it is a visual observation (otherwise is TextualMetadata_ or StressMetadata_) plus a unique sigmoid value.

Additionally to the CQs, we performed a validation procedure in order to inspect the syntactic and structural quality of the metadata in the KG and to check the consistency of them. Abiding by the closed-world criteria, custom SHACL consistency checking rules and native ontology consistency checking, such as OWL 2 DL reasoning, were utilized. By using the first, one can discover constraint breaches such as cardinality contradictions or imperfect/missing information. By using the latter, the semantics at the terminological level, known as TBox, are taken into consideration as validation like in the occasion of class disjointness. An exemplary shapes constraint is shown below which unfolds a constraint which dictates that all targeted instances of the class ‘InformationOfInterest’ will always have exactly one string value in their datatype property ‘:hasType’.

The consistency of the KG was found adequate, as out of 56 SHACL rules, from which 21 referred to object type properties and 35 to data type properties, none of them returned any invalidation of the rule. Moreover, we checked if there exist instances which belong to intersection of classes, as we did not desire such a case, and there were not any.

4.2. Points of interest management mechanism evaluation

The evaluation of the POI management mechanism was performed using the usual precision, recall, and F1-score used for information extraction systems (Equations 1, 2, and 3), for the creation or update of POIs from visual and textual messages. We followed this direction as the POI management mechanism can be regarded as an information extraction mechanism, where a query is received (a message from the textual, visual, or stress analysis components), and some information is extracted from the KG (a POI is created or updated).

(1) \begin{equation} precision = \frac{|\{Relevant Instance\}\cap\{Retrieved Instance\}|}{|\{Retrieved Instance\}|}\end{equation}

(2) \begin{equation} recall = \frac{|\{Relevant Instance\}\cap\{Retrieved Instance\}|}{|\{Relevant Instance\}|}\end{equation}

(3) \begin{equation} F1 = 2*\frac{recall*precision}{recall + precision}\end{equation}

Retrieved Instances are considered all the visual (or textual) messages for which the POI management mechanism, did not return an error when we casted a message in order to create or update a POI.

Relevant Instances are considered all the visual (or textual) messages for which the POI management mechanism, managed to create or update a POI, when we casted a message with them.

We denote by $Retrieved_t$ and $Retrieved_v$ the number of retrieved textual and visual messages, respectively. $Relevant_t$ and $Relevant_v$ are the numbers of relevant textual and visual messages, respectively. Next, $precision_t$ and $precision_v$ are the precision scores for the textual and visual messages, $recall_t$ and $recall_v$ are the recall scores for the textual and visual messages, and $F1_t$ and $F1_v$ are the F1 scores for the textual and visual messages, respectively.

The dataset on which we evaluated our POI management mechanism contains a set of 1201 textual messages and 600 visual messages, and can be found here footnote 1. Notice that in order to tackle potential biases, the value of each label in each message was randomly collected from a gold standard dataset created from domain experts. Interestingly, all the messages (either textual or visual) were considered Retrieved, meaning that our POI management mechanism did not return any error, for any given message both for textual and visual. Thus, $Retrieved_t = 1201$ and $ Retrieved_v = 600$ . The same does not hold for the relevant messages, for either textual or visual, as for textual the $Relevant_t$ messages were 1057, and for visual the $Relevant_v$ messages were 534.

Based on the aforementioned numbers the precision, recall, and F1-scores for both textual and visual messages can be found in Table 6. Notice, the results are rounded to four decimals.

Table 6. Precision, Recall, and F1-scores for textual and visual messages

4.3. Decision support system

In order to validate the Visual and Textual Analysis message severity assessment of the DSS, standard metrics like precision, recall, F1-score, and Accuracy score were used. The annotated dataset for the Visual DSS module was split at 76 entries for training and 20 entries for testing using cross validation (10-folds). In order for the accuracy scores and the rest of the metrics to be comparable among the two different sub-models of DSS, in the testing part of the Textual DSS we used 20 entries as well. The four different ML algorithms were tested after hyperparameter tunning for each one. In the following Table 7, we can see the results for each one of the models according to their best achieved accuracy score.

Table 7. Machine Learning algorithms performance according to Accuracy score

The best performing algorithm for the Visual Analysis module is DT Classifier with $max\_depth = 5$ (the maximum depth of the tree), $max\_leaf\_nodes = 10$ (the maximum amount of nodes that can be used when growing a tree in best-first fashion) and $random\_state = 42$ (controls the randomness of the estimator – if the $random\_state$ is an integer a deterministic behavior during fitting is obtained). In the following Table 8, we can see more analytically the validation metrics for DT (see Figure 10).

In the following figure, there is the confusion matrix of the testing phase of DT algorithm, with the best performing parameters.

Table 8. Precision, Recall, F1-score for DT algorithm

The best performing algorithm for Textual Analysis DSS is SVM with $C=10$ (Regularization parameter), $gamma=0.1$ (Kernel coefficient for ‘rbf’, ‘poly’, and ‘sigmoid’), and $random\_state=42$ . In the following Table 9, we can see more analytically the validation metrics for SVM. Following there is the confusion matrix of the testing phase of the SVM algorithm with the best performing parameters (see Figure 11).

Table 9. Precision, Recall, F1-score for SVM algorithm

Figure 10. Confusion matrix of DT algorithm.

Figure 11. Confusion matrix of SVM algorithm.

4.4. Task creation mechanism evaluation

The task creation mechanism was evaluated through a use case scenario. More specifically, we took data from AAWA8, for the water levels of the river Bacchiglione in the urban area of Vicenza. AAWA has installed across the river 41 different sensors that measure the water level of the river, and if the water level exceeds a threshold then a warning message is triggered. The threshold varies from sensor to sensor as the geography of each area around the sensor is different, and a water level that is allowed for one sensor might not be allowed for another.

Based on this setup, we took data of water level measures concerning a period of more than a decade, from 2010 till 2022. In this dataset, there were $\sim$ 50 violations of the thresholds across all sensors. For these violations, our DSS mechanism correctly created a message for tasks that need to be performed, and with the help of the XR4DRAMA framework these tasks were attached to POIs.

4.5. Danger zone mechanism evaluation

For the evaluation of the danger zone management mechanism, we followed a direction similar to the evaluation of the POI management mechanism (see Section 4.2). The evaluation of the danger zone was performed using the usual precision, recall, and F1-score used in information retrieval systems (Equations 1, 2, and 3), for the creation or update of danger zones from DSS messages. We followed this direction, as the danger zone management mechanism can be regarded as an information retrieval mechanism, where a query is received (a message from the DSS), and some information is retrieved from the KG (a danger zone is created or updated).

Retrieved Instances are considered all the DSS messages for which the danger zone management mechanism, did not return an error when we casted a message in order to create or update a danger zone.

Relevant Instances are considered all the DSS messages for which the danger zone management mechanism, managed to create or update a danger zone, when we casted a message with them.

We denote by $Retrieved_{DSS}$ the number of retrieved DSS messages. $Relevant_{DSS}$ are the numbers of relevant DSS messages. Next, $precision_{DSS}$ are the precision scores for the DSS, $recall_{DSS}$ are the recall scores for the DSS messages, and $F1_{DSS}$ are the F1 scores for the DSS messages.

The dataset on which we evaluated our danger zone management mechanism contains a set of 2400 DSS messages and can be found here footnote 3. Notice that in order to tackle potential biases, the value of each label in each message was randomly collected from a gold standard dataset created from domain experts. The $Retrieved_{DSS}$ messages were 2344 and the $Retrieved_{DSS}$ were 2068.

Based on the aforementioned numbers the precision, recall, and F1-scores for both textual and visual messages can be found in Table 10. Notice that the results are rounded to four decimals.