Table of Contents

    Topic review

    Evaluation of Digital Applications

    View times: 294
    Submitted by: Paul Weigel


    Digitalization is a transformation process which has already affected many parts of industry and society and is expected to yet increase its transformative speed and impact. In the energy sector, many digital applications have already been implemented. However, a more drastic change is expected during the next decades. Good understanding of which digital applications are possible and what are the associated benefits as well as risks from the different perspectives of the impacted stakeholders is of high importance. On the one hand, it is the basis for a broad societal and political discussion about general targets and guidelines of digitalization. On the other hand, it is an important piece of information for companies in order to develop and sustainably implement digital applications. This entry suggests a methodology to holistically analyze digital applications in the energy sector. The intended purpose of the suggested methodology is to provide a complexity-reduced fact base as input for societal and political discussions and for the development of new digital products, services, or business models. While the methodology is outlined in this entry, in a follow-up article the application of the methodology will be presented and the use of the approach reflected.

    1. Introduction

    Digitalization has the potential to have a drastic impact on all steps of the energy value chain as well as on all stakeholders, including the environment, the overall society, and the national economy. Good knowledge of the functionality of digital applications and their impact on all stakeholders is necessary. On the one hand, it enables a broad societal and political discussion to set the general targets and guidelines for the digitalization process and on the other hand, it is an important piece of information for companies to develop and sustainably implement digital applications.

    Therefore, a potential methodology to holistically assess and evaluate digital applications in the electricity sector is suggested. While the methodology is outlined in this entry, a follow-up article will present a detailed description, an applied use case of the methodology and a critical reflection of the approach.

    2. Suggested Evaluation Methodology for Digital Applications

    A variety of conventional as well as rather recent methods, such as living labs and design thinking, are already applied in the development of new products, services, and business models. However, a holistic analysis of benefits and potential risks of digital applications, taking into account all impacted stakeholders, provides an effective way to identify and solve potential impediments and drawbacks early on.

    The following suggested methodology is presented as an outlook in [1]. It offers a basic framework to perform a holistic analysis of technical, ecological, social and economic aspects of digital applications. A combination of a multi-criteria analysis (MCA), a life cycle assessment (LCA), and semi-structured interviews is suggested. In the following figure, the overall concept and basic functionality of the combination of methodologies is depicted. The concept can be adapted to the diverse kinds of digital applications and different analysis depths, and it allows the inclusion of quantitative as well as qualitative criteria. The result is an overall evaluation with reduced complexity including specific highlights, risks and respective solution options.

     Suggested combination of methodologies for holistic assessment and evaluation of digital applications.

    As depicted in the above figure, the basic structure for the evaluation is a multi-criteria analysis. However, the assessment of the different criteria defined in the MCA is conducted either within the life cycle assessment or via the semi-structured expert interviews. This offers the flexibility to either perform a more qualitative “Quick Check” based mostly on expert interviews (or even literature review) or if required a qualitative “Deep Analysis” via the LCA.

    2.1 Multi-criteria analysis (MCA)

    The MCA is a tool to assess different alternatives with high complexity, high uncertainty, inhomogeneous criteria, inhomogeneous availability of information and multiple perspectives / stakeholders. Diverse assessment criteria can be evaluated on a common basis, and a compensation between criteria is possible. Therefore, the complexity of the result can be reduced, which is a required prerequisite for broader social or political discussions. Furthermore, weighting factors can be used to reflect different (stakeholder) perspectives.

    A general description of the MCA methodology is given for example in [2][3][4][5]. The application of the MCA in the energy sector and for sustainability assessment is described in [6][7][8][9][10] and [11].

    2.2 Life cycle assessment (LCA)

    The life cycle assessment is mainly a tool to assess the ecological (and social) sustainability of products and services along the entire life cycle, i.e. cradle-to-grave. Furthermore, it can help to identify risks and bottlenecks and provide a basis to identify potential solutions. The cradle-to-grave approach, including the supply of primary materials, production, transport, use-phase and recycling and/or landfilling ensures the integration of factors which are not reflected in purely economically driven methods and thereby avoids that the related potential impacts and costs are missed.

    In the context of the proposed combination of methods the LCA is used to assess mainly technological and ecological but also economic and social aspects which are mostly quantifiable and are first-order effects (direct effect e.g. use of resources for hardware) and directly attributable effects of second-order (e.g. direct reduction of energy consumption due use of application).

    The LCA is defined by  [12] and [13].  Literature on the general application of LCA is for example given by [14][15][16][17][18][19]. The application of LCA in the energy sector is described for example in [20][21][22][23]. How the LCA can be used to assess social aspects is described in [24].

    2.3 Semi-structured Interviews

    While the LCA is used to assess mostly quantifiable first-order and, to a certain extent, second-order effects, the semi-structured interviews are used when information availability or reliability is poor. Therefore, interviews are mostly applicable for third-order effects, e.g. rebound effects as well as some second-order effects, which cannot be attributed easily to the digital application under assessment, e.g. effects, which occur in the larger energy system and are also influenced by a variety of other factors. In the interview, the identified criteria are evaluated based on the logic defined in the MCA. The methodological background of expert interviews is described in [25].

    A first test of the described methodology is currently being conducted for the German “smart meter” roll-out in cooperation with “smart meter” manufacturers. While an overall evaluation result cannot yet be given, first risks/bottlenecks have already been identified. One risk is the use of conflict (e.g., tin, tantalum) and scarce/critical (e.g., rare earth elements) materials. However, this risk is not specific to “smart meters” but applies to most modern communication devices. Besides that, some manufacturers of “smart meters” have managed to minimize or even eliminate the use of certain conflict and scarce materials. Another risk is a potential rebound effect. Although field tests have shown a reduction in energy consumption after the implementation of “smart meters”, it is unknown to what extent these reductions are permanent. A lower reduction of energy demand could offset the life cycle green house gas (GHG) balance. This risk could be met in the future with automated demand response mechanisms for “smart” household devices based on, for example, price signals. Furthermore, missing standards such as for “smart home” network communication protocols and interfaces could lower the implementation speed of applications and thereby reduce overall benefits.

    Full application of the methodology and discussion of the results, including an evaluation of the suitability of the methodology is yet to come. However, the first experiences of the test suggest that the proposed methodology as such could be suitable for the purpose of a holistic evaluation of digital applications.

    3. Conclusions

    Most digital applications do not only cause benefits but also have risks of downsides. Therefore, a good understanding of both benefits and risks from all stakeholders’ perspectives at an early stage of development is essential to find solutions to mitigate the risks and make full use of the benefits. A methodology for a holistic assessment and evaluation of digital applications in the energy sector is presented. The methodology consists of a combination of multi-criteria analysis (MCA), life cycle assessment (LCA), and semi-structured expert interviews. Going forward the suggested methodology needs to be further detailed, tested and revised. It could also be adapted to be suitable for digital applications in other, non-energy sectors. Overall, the aim is to create an evaluation basis that provides a structured approach to holistically assess digital applications and provide assessment results with reduced complexity.

    The publication can be found here:


    1. P. Weigel and M. Fischedick, “Review and Categorization of Digital Applications in the Energy Sector,” Applied Sciences, vol. 9, no. 24, 2019, doi: 10.3390/app9245350.
    2. S. Greco, M. Ehrgott, and J. R. Figueira, Eds., Multiple Criteria Decision Analysis, vol. 233. New York, NY: Springer New York, 2016.
    3. M. Velasquez and P. Hester, “An analysis of multi-criteria decision making methods,” International Journal of Operations Research, vol. 10, pp. 56–66, May 2013.
    4. Department for Communities and Local Government, Multi-criteria analysis: a manual. Wetherby: Communities and Local Government, 2009.
    5. C. Zopounidis and P. M. Pardalos, Eds., Handbook of multicriteria analysis. Heidelberg: Springer, 2010.
    6. J. R. San Cristóbal Mateo, Multi Criteria Analysis in the Renewable Energy Industry. Springer Science & Business Media, 2012.
    7. A. Kumar et al., “A review of multi criteria decision making (MCDM) towards sustainable renewable energy development,” Renewable and Sustainable Energy Reviews, vol. 69, pp. 596–609, Mar. 2017, doi: 10.1016/j.rser.2016.11.191.
    8. H. Polatidis, D. A. Haralambopoulos, G. Munda, and R. Vreeker, “Selecting an Appropriate Multi-Criteria Decision Analysis Technique for Renewable Energy Planning,” Energy Sources, Part B: Economics, Planning, and Policy, vol. 1, no. 2, pp. 181–193, Jul. 2006, doi: 10.1080/009083190881607.
    9. I. Wilkens, “Multikriterielle Analyse zur Nachhaltigkeitsbewertung von Energiesystemen - Von der Theorie zur praktischen Anwendung,” 2012.
    10. I. Braune, A. Pinkwart, and M. Reeg, “Application of Multi-Criteria Analysis for the Evaluation of Sustainable Energy Systems—A Review of Recent Literature,” Proceedings of 5th Dubrovnik Conference on Sustainable Development of Energy, Water and Environment Systems, Sep. 2009.
    11. M. Cinelli, S. R. Coles, and K. Kirwan, “Analysis of the potentials of multi criteria decision analysis methods to conduct sustainability assessment,” Ecological Indicators, vol. 46, pp. 138–148, Nov. 2014, doi: 10.1016/j.ecolind.2014.06.011.
    12. ISO/TC 207/SC 5, “ISO-14040:2006 Environmental management — Life cycle assessment — Principles and framework.” 2006.
    13. ISO/TC 207/SC 5, “ISO-14044:2006 Environmental management — Life cycle assessment — Requirements and guidelines.” 2006.
    14. A. Azapagic, “A Life Cycle Approach to Measuring Sustainability,” in Encyclopedia of Sustainable Technologies, M. A. Abraham, Ed. Oxford: Elsevier, 2017, pp. 71–79.
    15. United States Environmental Protection Agency, “Life Cycle Assessment: Inventory Guidelines and principles,” Scientific Applications International Corporation (SAIC), Feb. 1993.
    16. United States Environmental Protection Agency, “Life Cycle Assessment: Principles and Practice,” Scientific Applications International Corporation (SAIC), Reston, USA, May 2006.
    17. M. Finkbeiner, Ed., Special Types of Life Cycle Assessment. Dordrecht: Springer Netherlands, 2016.
    18. M. Z. Hauschild, R. K. Rosenbaum, and S. I. Olsen, Eds., Life Cycle Assessment. Cham: Springer International Publishing, 2018.
    19. M. Z. Hauschild and M. A. J. Huijbregts, Eds., Life Cycle Impact Assessment. Dordrecht: Springer Netherlands, 2015.
    20. B. A. Benedict, “Understanding Full Life-cycle Sustainability Impacts of Energy Alternatives,” Energy Procedia, vol. 107, pp. 309–313, Feb. 2017, doi: 10.1016/j.egypro.2016.12.158.
    21. S. S. van Dam, C. A. Bakker, and J. C. Buiter, “Do home energy management systems make sense? Assessing their overall lifecycle impact,” Energy Policy, vol. 63, pp. 398–407, Dec. 2013, doi: 10.1016/j.enpol.2013.09.041.
    22. G. G. Sias, “Characterization of the Life Cycle Environmental Impacts and Benefits of Smart Electric Meters and Consequences of their Deployment in California,” UCLA, 2017.
    23. R. Turconi, A. Boldrin, and T. Astrup, “Life cycle assessment (LCA) of electricity generation technologies: Overview, comparability and limitations,” Renewable and Sustainable Energy Reviews, vol. 28, pp. 555–565, Dec. 2013, doi: 10.1016/j.rser.2013.08.013.
    24. S. S. Muthu, Ed., Social Life Cycle Assessment. Singapore: Springer Singapore, 2015.
    25. A. Bogner, Ed., Das Experteninterview: Theorie, Methode, Anwendung. Opladen: Leske + Budrich, 2002.