Categories
Research

Design and Realization of Alzheimer’s Artificial Intelligence Technologies

AAIT is a management system for caregivers that employs artificial intelligence (AI) and Internet of things (IoT) devices to increase the independence and mobility of Alzheimer’s patients while also establishing safety measures. Recognized by Marvel Studios and published in the proceedings of the 6th IEEE International Conference on Future Internet of Things and Cloud.

Overview

After reviewing current technologies that assist the everyday lives of those living with Alzheimer’s, we recognized two problems: cost and accessibility. To address these issues, we designed and developed a management system for caregivers and patients that employs artificial intelligence (AI) and Internet of things (IoT) devices to increase the independence and mobility of Alzheimer’s patients while also establishing safety measures.

My Role

  • Design: Design solution architecture and program logic
  • Prototype: Create a working prototype of the AAIT system using Workflow and IFTTT

Problem (Background)

Alzheimer’s is an irreversible, progressive brain disorder that affects memory, behavior, and thinking. Based on Alzheimer’s Disease International’s report, nearly 44 million people worldwide have Alzheimer’s or a related dementia. In 2017, 5.5 million Americans were living with Alzheimer’s disease (Alzheimer’s Association). Alzheimer’s is the sixth leading cause of deaths in the United States. Symptoms of Alzheimer’s include wandering and memory loss. Alzheimer’s is progressive, meaning the symptoms typically worsen over time, and each patient’s case varies in degree. Furthermore, dementia places an emotional and financial stress on family caretakers. In 2018, Alzheimer’s and other dementias will cost the nation $277 billion dollars. By 2050, costs could reach up to $1.1 trillion (“2018 Alzheimer’s disease facts and figures,”Alzheimer’s & Dementia). With this in mind, an inexpensive solution could impact a greater audience while addressing common symptoms of Alzheimer’s.

As the disease does not have a cure at this time, efforts have been focused on technological developments that could assist the everyday lives of those with Alzheimer’s. Electronic devices have been designed to assist caretakers in locating the missing patient with Alzheimer’s (“Evaluating an electronic monitoring system for people who wander,” American Journal of Alzheimer’s Disease). Furthermore, new technologies such as electronic tagging, assistive technology, and interventions meet the psychosocial and safety needs of Alzheimer’s patients. The advent of new technologies provides a solution to symptoms including wandering and forgetfulness (“Technology and personhood in dementia care,” Quality in Ageing and Older Adults; “Supporting safe walking for people with dementia: User participation in the development of new technology,” Gerontechnology; “The use of global positional satellite location in dementia: a feasibility study for a randomised controlled trial,” BMC Psychiatry), but these devices are limited and not cost-effective (“Intelligent Assistive Technology Applications to Dementia Care: Current Capabilities, Limitations, and Future Challenges,” The American Journal of Geriatric Psychiatry).

Goal

This project aims to develop a high technology solution to improve the lives of Alzheimer’s patients while keeping costs and feasibility in mind. 

Solution Architecture

While building this project, we realized two things: first, technology is used everywhere, e.g. smartphones; second, these existing technologies can be connected to create a system to assist the everyday lives of Alzheimer’s patients.

The AAIT solution offers a platform for caregivers to register patients’ information, track patients’ movements, define warning zones, and receive notifications when a patient moves out of a set of predefined zones. In addition, the AAIT solution can automatically trigger actions to control the related Internet of Things (IoT) devices to save energy or provide convenience for the patients.

  • Patient's GPS location triggers an alert to the caregiver and controls whether the lights in the house are on or off. The patient's movement information is saved to the cloud.
  • The GPS location in the patient's phone controls the lights in the patient's house. When the patient leaves the house, the lights will turn off. When the patient enters the house, the lights will turn on.
  • The GPS location in the patient's phone is used to send automatic alerts to the patient's caregiver. An alert will be sent out via SMS or phone call depending on the deemed severity of the patient's movement.
  • All of the patient's movement data will be stored to the cloud for further analysis.

As shown in Solution: AAIT System Overview, there are two kinds of role players in the system. The first role player is the caregiver who provides care services to the patient. The second role player is the patient who receives the care services provided by the caregiver. In this context, the caregiver is a service provider, and the patient is a service consumer. The services are conducted through a series of interactions between service providers and service consumers to achieve a certain business goal or provide a solution for addressing one or more pinpoints. The IoT devices are used as assisting tools or equipment to get sensing data and trigger actions. All the interactions and behaviors are captured and stored in a database to form a big data set for the AAIT system.

As an example solution scenario, the patient’s device or system can alert the caregiver if a certain action has occurred. For example, when the patient leaves home or a living place, an alert message is automatically sent to the caregiver. Meanwhile, the room lights are turned off to save energy while the patient is gone.

The overall solution architecture for the AAIT system adopts a Service-Oriented Architecture (SOA) with a focus on a data-driven framework. All the key data sets generated from the caregivers, patients, and IoT devices are automatically captured and saved to the cloud. The workflow of enabling the interactions between caregivers and patients is also connected with the data sets in the cloud.

Once the data sets are collected, they can be analyzed using big data processing technologies and artificial intelligence algorithms to decode the insights behind the data sets captured in the AAIT system.

Prototype

Materials

From the hardware perspective, this project needs two smartphones for the caregiver and patient respectively. As an example device of Internet of Things, a smart light bulb is used in the proposed AAIT system. The software suite includes Google Drive for storing data sets, Dropbox as an example cloud storage, IFTTT as a service connector, and Workflow as an application running on smartphones to implement our solution scenarios.

Registration Workflow

When a new patient is enrolled in the AAIT system, the caregiver uses the following steps to capture some basic information for the Alzheimer’s patient. This paper designs a registration flow entitled Register in the Workflow app on iPhone to interactively record and capture patient data.

When the Register workflow is launched, the caregiver is prompted with questions about the patient, including name, blood type, and allergies.

As shown in the figure, once the Register workflow is launched, it will capture the current location automatically. The caregiver inputs the patient’s name and blood type, chooses all allergies, and takes the patient’s photo using the smart phone’s camera, and then saves it to the local phone’s Album and Dropbox for cloud access. Next, Trigger IFTTT Applet is used to pass the variables (Location, Current Date, Photo_Link, BloodType, and Allergy) to a Google Sheet stored on the cloud-based Google Drive.

The Workflow app Register can be used by a caregiver to capture the basic information when a patient is enrolled in the care service. It is noted that the Workflow app Register can serve as an extensible platform for the caregivers or IT administrators to add more functions.

Scenarios

Three demo scenarios: 1) When the patient enters the home safety zone, the caregiver receives an SMS notification, the smart lights in the house turn on, and the time and location of the patient's movement is saved in a Google Spreadsheet. 2) When the patient exits the safety zone, the caregiver receives an SMS notification, the smart lights in the house turn off, and the time and location of the patient's movement is saved in a Google Spreadsheet. 3) When the patient exits the warning zone, the caregiver receives an SMS notification and a phone call, the smart lights in the house remain off, and the time and location of the patient's movement is saved in a Google Spreadsheet.
Three scenarios showing how the patient’s movement event triggers notifications to the caregiver and controls smart home devices. Data from the patient’s movement event is captured and stored in the cloud.

Building Demo Scenarios using IFTTT

IFTTT uses an if this, then that logic; Event triggers in the demo scenarios include enter home, exit safety zone, and exit warning zone. These events trigger actions such as turn on/off lights, send an SMS to the caregiver, call the caregiver, and upload patient movement data to Google Spreadsheets.
IFTTT Logic
  • The Alert System uses the following logic: if Event, then Action. Depending on the trigger (the patient's location), the system sends an appropriate notification to the caregiver (either via SMS or phone call).
  • The Big Data Collector uses the following logic: if Event, then Action. When the patient exits or enters certain zones, the movement data is captured and stored in the cloud.
IFTTT Building Interface Sample

Features

Internet of Things Services

Internet of things (IoT) has been widely used for connections of devices, interactions between devices and people, and device-to-device communications (“A deep learning approach for condition-based monitoring and fault diagnosis of rod pump,” Services Transactions on Internet of Things (STIOT)), (“Trust as a service for SOA- based IoT systems,” Services Transactions on Internet of Things (STIOT)). In our proposed AAIT system, we use the interaction layer, business scenario layer, service layer, component layer, and infrastructure layer as the initial system architecture. Some detailed architecture building blocks in these five layers and the additional layers can be added incrementally in the future.

The GPS location in the patient's phone controls the lights in the patient's house. When the patient leaves the house, the lights will turn off. When the patient enters the house, the lights will turn on.

In the AAIT system, smartphones are used to install applications, which communicate with IoT devices and interact with people including caregivers and Alzheimer’s patients. As shown in the illustration diagram, the patient’s location triggers the smart light bulb in the house to turn either on or off based on the patient’s location. The patient’s location service is installed on his/her smartphone. The smart light bulb is an IoT device.

In the AAIT system, the patient’s location is a key triggering data source for the smart light bulb. From a solution architecture perspective, the smart light bulb is part of the IoT infrastructure layer. Triggering on or off is part of the IoT Reusable Service Layer. The patient’s location data is associated with triggering decisions such as sending alerts to the caregiver or turning on or off the smart light bulb. Sending alerts and triggering IoT devices are integrated to support various solution scenarios in the AAIT system.

It is noted that the patient’s data can be extended from location to any type of contents captured by the patient’s smartphone. The patient’s data not only controls the light bulbs on and off, but also can be used to connect any Internet-enabled devices such as refrigerators, air conditioners, service robots, and cameras.

Alert Services

The GPS location in the patient's phone is used to send automatic alerts to the patient's caregiver. An alert will be sent out via SMS or phone call depending on the deemed severity of the patient's movement.

Alert services are the key communication mechanisms that the caregiver uses to get notifications when special events occur based on predefined rules. As illustrated, when the patient reaches Enter Home, a short message service (SMS) is used to alert the caregiver via the AAIT system. When the patient reaches Exit Safety Zone, a short message service (SMS) is used to alert the caregiver. When the patient moves out further to Exit Warning Zone, a short message service (SMS) and Phone Call service are used concurrently to alert the caregiver via the AAIT system.

The alert services in the AAIT system can be expanded to email, instant message, alarm, artificial intelligence-powered speaker, and large screen display.

Big Data Collection

In the AAIT system, there are three categories of data to be collected for building intelligence and insights which can be used to support the Alzheimer’s patients to get better and more timely services. The first category of data is related to the basic information about Alzheimer’s patients. Some basic information includes, but not limited to, a patient’s name, address, contact, blood type, and other behaviors captured before. The second category of data includes all movement behaviors (time, location, leaving home event, etc.) for a patient. The third category of data includes triggering events, interactions with caregivers, and control commands for the IoT devices.

As a scalable resource sharing platform, cloud computing has been widely used in distributed systems (“A new access control scheme for protecting distributed cloud services and resources,” Services Transactions on Cloud Computing (STCC)). In the AAIT system, the patient’s data collection (e.g., movement data, basic information, etc.) is taken from a smartphone or other device and is saved to a storage space on a cloud platform.

Artificial Intelligence: Patient Data Analysis

The AAIT System gradually captures a patient’s real-time data when the three scenarios occur.

Since all movement behaviors have been stored for the Alzheimer’s patients, Google Spreadsheets is used to store and check the event history with the associated patient’s name, event name, the occurred time, location address, and a hyperlink to the location address to open easily in navigation.

Google Explore is an artificial intelligence tool created by Google. It analyzes the data stored in the Google Spreadsheet. In addition, it monitors what a user sees, types, and interacts with instantly.

Google Explore is used in this solution to leverage machine learning to analyze data. It acts as a software robot to help find out the movement patterns of the Alzheimer’s patient.

This robot can answer questions based on the contents. In the AAIT system, the software robot Explore can answer the following example questions such as the most frequent action, Count of Time for each Action, and unique values in Action.

The built-in Natural Language Processing (NLP) technologies make Google Explore smarter. In the proposed AAIT System, the Explore can be used to list all the unique patients’ names and the most frequent locations, as well as to visualize the data in a chart using questions in natural language.

Conclusions

It is costly to create devices that can aid the everyday lives of Alzheimer’s patients. Our proposed approach, the AAIT system, is a feasible approach to leverage the latest information technologies including Internet of Things, cloud, big data, and artificial intelligence to help the caregivers improve their efficiency, effectiveness, and smartness in a cost-effective manner.

Future Directions

In the future, the AAIT system can be expanded and embedded into a wearable device. For example, the tracking device can be embedded into a smartwatch or necklace instead of using the GPS location tracking of the patient’s smartphone. This would minimize the possibility that the patient forgets the device that the tracker is located in, which is an important component for the AAIT system to function properly. As another example, integrated with augmented reality (AR), the AAIT system can be further adapted to the needs of the patients. The AR technology seamlessly blends digital elements with the real- world view so when Alzheimer’s patients put on AR-powered AAIT glasses, they are immediately immersed in a user-adapted and customized environment with features including facial and voice recognition of loved ones and notifications to assist medication management.

Achievements

I would like to thank the Walt Disney Studios Motion Pictures, Marvel Studios, Synchrony Bank, Dolby Laboratories, Broadcom MASTERS, and the American Association for the Advancement of Science for sponsoring Marvel Studios’ Superpower of STEM Challenge, which chose this project as one of five finalists in 2017.

Research Paper

This project was detailed in the paper titled “Design and realization of Alzheimer’s Artificial Intelligence Technologies (AAIT) system,” which was published in the proceedings of the 6th IEEE International Conference on Future Internet of Things and Cloud (FiCloud 2018), Barcelona, Spain, 141-148, August 2018.