Technologies and Applications for big data value

Dr Edward Curry

Co-Principal Investigator- the Insight Centre for Data Analytics, funded investigator- LERO The Irish Software Research Centre, Ireland

Computer science was created by humankind to solve problems. In 100 BC, early hand-powered computing devices such as the Antikythera mechanism were designed to calculate astronomical positions. In the 1800s, Charles Babbage proposed the Analytical Engine to solve general-purpose computational tasks. In the 1900s, the Bomde by Turing and Welchman was critical to code-breaking. Advances in computer science have been driven by the need for humanity to solve the most pressing challenges of the day. Today, computer science tackles significant societal challenges like organising the world’s information, personalised medicine, the search of the Higgs boson, climate change, and weather forecasts.

This book aims to educate the reader on how recent advances in technologies, methods, and processes for big data and data-driven Artificial Intelligence (AI) can deliver value to address problems in real-world applications. The book explores cutting-edge solutions and best practices for big data and data-driven AI and applications for the data-driven economy. It provides the reader with a basis for understanding how technical issues can be overcome to offer real-world solutions to major industrial areas, including health, energy, transport, finance, manufacturing, and public administration.

The book’s contributions emanate from the Big Data Value Public-Private Partnership and the Big Data Value Association, which have acted as the European data community’s nucleus to bring together businesses with leading researchers to harness the value of data to benefit society, business, science, and industry. The technological basis established in the BDV PPP will seamlessly enable future Partnership on AI, Data, and Robotics.

The book is of interest to two primary audiences: first, undergraduate and postgraduate students and researchers in various fields, including big data, data science, data engineering, and machine learning and AI. Second, practitioners and industry experts engaged in data-driven systems and software design and deployment projects who are interested in employing these advanced methods to address real-world problems.

This book is arranged in two parts. The first part contains “horizontal” contributions of technologies and methods which can be applied in any sector. The second part includes contributions of innovative processes and applications within specific “vertical” sectors. First chapter provides an overview of the book by positioning the chapters in terms of their contributions to technology frameworks, including the big data value reference model and the AI, Data and Robotics framework, which are key elements of the Big Data Value Public-Private Partnership and the upcoming Partnership on AI, Data, and Robotics.

Part I: Technologies and Methods details key technical contributions which enable data value chains. Second chapter investigates ways to support semantic data enrichment at scale. The trade-offs and challenges of serverless data analytics are examined in third chapter. Benchmarking of big data and Artificial Intelligence (AI) pipelines is the objective of fourth chapter, while fifth chapter presents an elastic software architecture for extreme-scale big data analytics. Sixth chapter details privacy-preserving technologies for trusted data spaces. Leveraging data-driven infrastructure management to facilitate AIOps is the focus of seventh chapter, and unified big-data workflows over High-Performance-Computing (HPC) and the cloud are tackled in eighth chapter.

Part II: Processes and Applications details experience reports and lessons from using big data and data-driven approaches in  processes and applications. The chapters are co-authored with industry experts and cover domains including health, law, finance, retail, manufacturing, mobility, and smart cities. Ninth chapter presents a toolkit for deep learning and computer vision over HPC and cloud architectures. Applying AI to manage acute and chronic clinical conditions is the focus of tenth chapter, while eleventh chapter explores 3D human big data exchange between the health and garment sectors. In twelfth chapter, we see how legal knowledge graphs can be used for multilingual compliance services in labour law, contract management, and geothermal energy. Thirteenth chapter focuses on big data analytics in the banking sector with guidelines and lessons learned from CaixaBank. Fourteenth chapter explores data-driven AI and predictive analytics for the maintenance of industrial machinery using digital twins. Fifteenth chapter investigates big data analytics in the manufacturing sector, and sixteenth chapter looks at the next generation of data-driven factory operations and optimisation. Large-scale trials of data-driven service engineering are covered in seventeenth chapter. Eighteenth chapter describes approaches for model-based engineering and semantic interoperability for digital twins across the product life cycle. In nineteenth chapter, a data science pipeline for big linked earth observation data is presented, and twentieth chapter looks ahead towards cognitive ports of the future. Distributed big data analytics in a smart city is the focus of twenty-first chapter, and twenty-second chapter looks at system architectures and applications of big data in the maritime domain. The book closes with twenty-third chapter exploring knowledge modelling and incident analysis for cargo.