It is a general principle to monitor the performance of a project against its project objectives. Whoever ‘examines and judges accomplishments and effectiveness [of an approach] is engaged in evaluation’ (Patton, 1990). Hence it is the overall goal of any research project to develop a suitable evaluation framework to judge the goal achievement with respect to scientific rigor as well as to practical applicability.
The success of the AEGIS project depends on effectively evaluating the AEGIS platform and then providing constant feedback to the AEGIS platform developers to further improve it. Hence the AEGIS project plans, executes, and evaluates AEGIS Data Value Chain Early Community Demonstrators from three different Public Safety & Personal Security (PSPS) domains, namely (1) Automotive, (2) Smart Home & Assisted Living, and (3) Insurance within WP5. These three AEGIS demonstrators will be complemented by further activities running independently to facilitate the generalization of the AEGIS platform to be utilized within other domains for different application scenarios, too. Obviously, these three demonstrators play a major role in the evaluation of the AEGIS platform, while using the data and the functionality of the AEGIS platform to create appropriate services in these three domains.
The main direct beneficiaries of the AEGIS platform are – in a more general perspective – the PSPS data scientists from three different sectors, automotive, smart home & assisted living, and insurance, as the three implementers of the data-driven AEGIS PSPS services, as well as the various stakeholders as PSPS users from three different sectors, who consume the AEGIS PSPS services. Hence, the satisfaction of those stakeholders with the AEGIS platform in general, with the demonstrator implementation process more specifically, as well as with the demonstrator adoption, has to be evaluated in a qualitative way.
One prerequisite for the successful evaluation of the AEGIS platform is that all required functionality for the execution of these three AEGIS Data Value Chain Community Demonstrators and beyond is provided through the AEGIS platform. Thereby it is required to capture the experiences of the stakeholders who have been using the AEGIS platform accordingly to develop the demonstrators. The AEGIS methodology implemented in the AEGIS platform has been derived from the AEGIS data value chain. The AEGIS Data Value Chain has been strongly motivated by the Big Data Value Chain developed by Curry et al. (2016). The AEGIS Data Value Chain is the foundation for enabling an AEGIS data to service process. It is required to capture the experiences of the stakeholders performing activities in these steps while implementing their demonstrators.
Due to the complexity of the AEGIS platform, the AEGIS evaluation framework to develop envisages the application of a mix of methods. Depending on a particular application goal, and a stakeholder group in focus of evaluating the goal achievement, a particular evaluation approach is suggested. A mix of semi-structured interviews, user surveys and test/evaluation cases is foreseen. The evaluation framework envisages the application of qualitative methods, due to the complexity of the AEGIS platform and the expected platform evaluation feedback. Some of the constructs derived from the state of the art analysis (e.g. quality dimensions, information systems success, system usefulness and usability to name three of them) will be used within the evaluation framework. These concepts are derived from the Information Systems (IS) discipline, having evolved relevant models and theories including Technology Acceptance Model (Davis 1989), Unified Theory of Acceptance and Use of technology (Venkatesh et al. 2003), Information Systems Success Model (DeLone and McLean, 2003), Integrated Model of User Satisfaction and Technology Acceptance (Wixom & Todd, 2005), Model for Digital Information Asset Evaluation (Viscusi and Batini, 2014), or the Expectation-Confirmation Model (Bhattacherjee, 2001) to name some of them.
Practically implementing the three AEGIS demonstrators is a continuous process. However each AEGIS demonstrator is expected to show progress throughout the three defined phases early, medium, and advanced. The tangible results of these phases constitute the demonstrator-specific milestones. The three AEGIS demonstrators are implemented by several PSPS data scientists in charge (from different applications domains) through executing demonstrator-specific test/evaluation cases. To capture the satisfaction of the users with the AEGIS platform, the evaluation framework suggests conducting ex-post interviews with those PSPS data scientists in charge of creating the demonstrators as well as ex-post interviews with PSPS demonstrator users.
Derived from the reviewed IS models, it is highly relevant for the evaluation framework to (1) capture the prior knowledge and experiences of the data scientists in charge, (2) capture the data scientists’ expectations and perceptions of the AEGIS platform in general, and finally (3) capture the data scientists’ expectations and perceptions of the AEGIS demonstrator specific development process in particular as the figure below suggests.
The following questions are appropriate to shed light on how PSPS data-scientists experienced the use of the AEGIS platform within the AEGIS project, taking also their prior experiences into account.
| Target | Example questions for guided interviews with PSPS data scientists |
| Capturing the knowledge and experiences of the PSPS data scientist | What knowledge and experiences have you already gained in (big) data-related activities and projects?
|
| Capturing the expectations of the data scientists as well as how the data scientist in charge perceived the AEGIS platform in general
|
From what you have so far learned in the application of the AEGIS platform, how useful do you perceive the AEGIS platform in general?
|
| Capturing the expectations of the data scientists as well as how the data scientist in charge perceived the AEGIS demonstrator development process using the AEGIS platform
|
How did you perceive the technical process of developing your AEGIS demonstrator with the AEGIS platform?
|
General example questions for PSPS data scientists on the AEGIS platform
The following example questions are appropriate to shed light on how PSPS users experience the use/consumption of the services provided by the AEGIS platform.
| Target | Example questions for guided interviews with PSPS users |
| Capturing the expectations of PSPS users as well as how they perceive the AEGIS demonstrator use phase | How do the services created by the AEGIS platform meet your specific needs?
|
General example questions for PSPS users on the developed services
The AEGIS platform is intended to enable a big data value chain (for public safety and personal security – PSPS), as defined in deliverable D1.1. Hence it should be a further goal of the evaluation framework to shed light on how successful this enabling process is from the perspective of PSPS data scientists as well as how satisfied involved PSPS data are with the support or the AEGIS platform for developing big data driven applications are, respectively.
The following questions are dedicated to be answered by PSPS data scientists, too. They are related to the AEGIS data value chain (as defined in D1.1, derived from Curry et al. 2016), while relevant functionality is implemented in the AEGIS platform. For each step of this big data value chain, a set of example questions is listed. The AEGIS platform can be seen as a tool targeted at data scientists to support all steps of the data value chain accordingly.
| AEGIS Data value chain | Example questions for interviews with data scientists |
| Data acquisition is the process of gathering, filtering and cleaning data, before any data analysis can be carried out (D1.1).
|
Does the AEGIS platform’s functionality intended for data acquisition meet your needs and expectations? If yes, how? If no, why not?
|
| Data analysis is concerned with making the raw data acquired amenable to use in decision-making as well as domain-specific usage (D1.1).
|
Does the AEGIS platform’s functionality intended for data analysis meet your needs and expectations? If yes, how? If no, why not?
|
| Data curation is the active management of data over its life cycle to ensure it meets the necessary data quality requirements for its effective usage (D1.1).
|
Does the AEGIS platform’s functionality intended for data curation meet your needs and expectations? If yes, how? If no, why not?
|
| Data storage is the persistence and management of data in a scalable way that satisfies the needs of applications (D1.1).
|
Does the AEGIS platform’s functionality intended for data storage meet your needs and expectations? If yes, how? If no, why not?
|
| Data usage covers the data-driven business activities that need access to data, its analysis, and the tools needed to integrate the data analysis within the business activity (D1.1). | Does the AEGIS platform’s functionality intended for data usage meet your needs and expectations? If yes, how? If no, why not?
|
General example questions for PSPS data scientists on the AEGIS platform
The successful implementation of the evaluation framework relies on a four step procedure:
- In a first step the data collection procedure will be started through involving key stakeholders from the AEGIS demonstrators into platform evaluation, by conducting interviews with PSPS data scientists who have executed their evaluation cases on how they perceived their use of the AEGIS platform.
- In a second step, the collected qualitative data will be analysed and then evaluation results will be derived, by performing a qualitative content analysis followed-up by summarizing and quantifying main statements from transcribed interviews.
- In a third step, the AEGIS platform developers will receive the evaluation results to help them to improve the quality, usability, and usefulness of the AEGIS platform, leading to a better meeting of needs, expectations and wants of the demonstrator users.
- In a fourth and final step, the evaluation input will be implemented into the AEGIS platform by the platform development team in a best possible way, if feasible.
This tight AEGIS project implementation plan allows the AEGIS evaluation framework to be applied in total three times within the AEGIS project runtime, as there are three different versions of the AEGIS early community demonstrators envisaged. However, the results of the final evaluation (at M30) cannot be fed back to the AEGIS platform developers within the project runtime. Software implementations or changes will then have to be conducted by the developers after project close out in the project exploitation phase.
Blog post authors: VIF