Sensor data and social networks

To investigate questions surrounding inclusion and diversity, it is helpful to have research methods that can offer a detailed and precise picture of how children interact with their peers and the direct environment. For this purpose, we develop and use sensor data driven approaches, which complement traditional methods such as field observations and questionnaires.

‍

Which sensor data can we use, and what can it tell us?

Our previous research has used GPS loggers to obtain children’s geographical locations, Bluetooth-based proximity tags to log face-to-face interactions between individuals, and multi-motion receivers (MMR) to measure physical activity levels.

Through multimodal analyses of this sensor data, we can monitor how children move within an environment, their contact with peers and their activities in the schoolyard during unstructured school breaks. Using machine learning algorithms, we can extract patterns representing social behaviors of children over space and time. This provides a method for automated monitoring, and social network analyses.

Now we are taking it a step further by developing a fully-automated method for measuring individual differences during play in the form of a smartwatch app. Not only can this simultaneously record social behavior, movements, locations, and heart rate via the built-in sensors, but it can also collect children's subjective experiences during play in real time. This so-called ecological momentary assessment (EMA) goes beyond measuring behavior to obtaining in-the-moment insights about children's thoughts and feelings. This is important, because behavioral data can not always be easily interpreted – for instance, when it is unclear whether a child is happy or sad to spend time in isolation. Overall, the new smartwatch system allows us to advance our understanding of social interactions among students, identify patterns of connection, stress, activity, and belonging, and address previous methodological limitations of GPS accuracy in indoor environments when analyzing spatial proximity.

‍

Automated monitoring

How can machine learning algorithms automatically identify different social behavioral patterns from spatiotemporal data? Recent developments in artificial intelligence enabled us to design an end-to-end machine learning framework to automatically learn and distinguish group behavior from spatiotemporal data. You could read more about this research (see key publications below).

‍

Social network analyses

How does physical space influence our social network? Using a spatiotemporal social network, we understand where children are spatially and socially positioned in their network, how strong their connections to the network is, how schoolyard features influence individuals’ social network and finally how such spatiotemporal networks change over time.

The “SmartPlay” smartwatch app

In collaboration with the eScience center (funding info), we further enhanced our sensing system and built the “SmartPlay” app, which is now being developed further in a team of experts in developmental psychology and human-media interaction (we can also be specific). Current development pathways include expanding the app’s pop-up question mechanism with adaptive follow-up and conditional questions, while simultaneously integrating Bluetooth-triggered questions to better understand the reasoning behind participants’ responses and to examine how specific building locations influence students’ sense of belonging. 

SmartPlay is a smartwatch app built for Android Wear OS that gathers real-time information about children’s play activities through a data-driven, interdisciplinary method. It is intended to advance research on play behavior, health, and related areas. Its main goal is to capture children’s personal experiences during play, offering meaningful insights into their actions and surroundings. The app is designed to help researchers integrate knowledge from different fields while collecting and analyzing data in an accessible and user-friendly manner. 

The SmartPlay Social Connectivity Demo (see below) visualises how parts of the SmartPlay system function, exploring the app’s visualization capabilities. In this case, the demo shows how Bluetooth signals can inform us about social connections between people. 

When multiple individuals are wearing a smartwatch, each watch will record its proximity to other watches based on how strong the Bluetooth signal is between them. The SmartPlay system can track these signal interactions over time and provide insights on how individuals were positioned in relation to each other, to the surrounding space, and view dynamic changes over time. Focusing on the data from one of the smartwatches, we can view:

• Left: spatial distribution of devices. How spatial movement and signal strength can be mapped together, combining Bluetooth with GPS information.
• Upper right: evolving network connections. How connectivity between devices varies dynamically by looking at signal strength levels over time.
• Bottom right: relative signal strength patterns. How relationships between watches (and their wearers) can be represented using Bluetooth connections.

People involved

Computer Science - Maedeh Nasri, Mitra Baratchi (Leiden University), Dennis Reidsma, Sina Ghorbani Kolahi (University of Twente).

Psychology - Carolien Rieffe, Adva Eichengreen, Brenda Sousa da Silva, Yung-Ting Tsou, Jiayin Zhao (University of Twente / Leiden University), Guida Veiga (University of Evora).

Architecture - Alexander Koutamanis (TU Delft).

Governance - Sarah Giest (Governance, Leiden University), Ellen Starke (School Alliances Amsterdam).

‍

Key publications

• Nasri, M., Tsou, Y.T., Koutamanis, A., Baratchi, M., Giest, S., Reidsma, D., & Rieffe, C. (2022). A novel data-driven approach to examine children’s movements and social behaviour in schoolyard environments Children, 9, 1177 (Special Issue: Child Psychomotricity: Development, Assessment, and Intervention). Doi: 10.3390/children9081177.
• Nasri, M., Fang, Z., Baratchi, M., Englebienne, G., Wang, S., Koutamanis, A., & Rieffe, C. (2023). A GNN-Based Architecture for Group Detection from Spatio-Temporal Trajectory Data In Advances in Intelligent Data Analysis XXI: 21st International Symposium on Intelligent Data Analysis, IDA 2023, Louvain-la-Neuve, Belgium, April 12–14, 2023, Proceedings (pp. 327-339). Cham: Springer Nature Switzerland.
• Veiga, G., Ketelaar, L., de Leng, W., Cachucho, R., Kok, J. N., Knobbe, A., Neto, C., & Rieffe, C. (2017). Alone at the playground European Journal of Developmental Psychology, 14, 44-61. doi: 10.1080/17405629.2016.1145111.
• Laanen, R., Nasri, M., van Dijk, R., Baratchi, M., Koutamanis, A., & Rieffe, C. (2023). Automated classification of pre-defined movement patterns: A comparison between GNSS and UWB technology. arXiv preprint arXiv:2303.07107 doi: 10.48550/arXiv.2303.07107.