How can we realize sophisticated purposeful movements? While agile motor control is now standard in robotics, the computational machinery supporting this capacity in biological organisms remains controversial. A theory called "active inference" assumes that actions arise by inferring limb trajectories from the desired effects in an exteroceptive (e.g., visual) space. Yet, how this theory affords efficient motor control is unclear. Our innovative solution rests on designing a hierarchical kinematic model that mimics the body structure and predicts the exteroceptive effects of movements. In particular, a simple but effective mapping from extrinsic to intrinsic coordinates is realized via inference, and it is then scaled up to drive complex kinematic chains. Rich goals can be specified in both domains using attractive or repulsive forces, allowing an agent to perform tasks like drawing a circle, reaching and avoiding objects. The proposed model reproduces sophisticated bodily movements and paves the way for computationally efficient and biologically plausible control of actuated systems.
Successful behaviour relies on the appropriate interplay between action and perception. The well-established dorsal and ventral stream theories depicted two distinct functional pathways for the processes of action and perception, respectively. In physiological conditions, the two pathways closely cooperate in order to produce successful adaptive behaviour. As the coupling between perception and action exists, this requires an interface that is responsible for a common reading of the two functions. Several studies have proposed different types of perception and action interfaces, suggesting their role in the creation of the shared interaction channel. In the present review, we describe three possible perception and action interfaces: i) the motor code, including common coding approaches, ii) attention, and iii) object affordance; we highlight their potential neural correlates. From this overview, a recurrent neural substrate that underlies all these interface functions appears to be crucial: the parieto-frontal circuit. Within this framework, the present review proposes a novel model that is inclusive of the superior parietal regions and their relative contribution to the different action and perception interfaces.
The motor disability due to stroke compromises the autonomy of patients and caregivers.To support autonomy and other personal and social needs, trustworthy, multifunctional, adaptive, and interactive assistive devices represent optimal solutions. To fulfil this aim, an artificial intelligence system named MAIA would aim to interpret users' intentions and translate them into actions performed by assistive devices. Analysing their perspectives is essential to develop the MAIA system operating in harmony with patients' and caregivers' needs as much as possible.Method: Post-stroke patients and caregivers were interviewed to explore the impact of motor disability on their lives, previous experiences with assistive technologies, opinions, and attitudes about MAIA and their needs. Interview transcripts were analysed using inductive thematic analysis.Results: Sixteen interviews were conducted with twelve post-stroke patients and four caregivers.Three themes emerged: 1) Needs to be satisfied, 2) MAIA technology acceptance, and 3) Perceived trustfulness. Overall, patients are seeking rehabilitative technology, contrary to caregivers needing assistive technology to help them daily. An easy-to-use and ergonomic technology is preferable.However, a few participants trust a system based on artificial intelligence.Conclusions: An interactive artificial intelligence technology could help post-stroke patients and their caregivers to restore motor autonomy. The insights from participants to develop the system depends on their motor ability and the role of patients or caregiver. Although technology grows exponentially, more efforts are needed to strengthen people's trust in advanced technology.
All our dexterous movements depend on the correct functioning of the network of brain areas. Knowing the functional timing of these networks is useful to gain a deeper understanding of how the brain works to enable accurate arm movements. In this article, we probed the parieto-frontal network and demonstrated that it takes 4 ms for the medial posterior parietal cortex to send inhibitory signals to the frontal cortex during reach planning. This fast flow of information seems not to be dependent on the availability of visual information regarding the reaching target. This study opens the way for future studies to test how this timing could be impaired in different neurological disorders.
We advance a novel active inference model of the cognitive processing that underlies the acquisition of a hierarchical action repertoire and its use for observation, understanding and imitation. We illustrate the model in four simulations of a tennis learner who observes a teacher performing tennis shots, forms hierarchical representations of the observed actions, and imitates them. Our simulations show that the agent's oculomotor activity implements an active information sampling strategy that permits inferring the kinematic aspects of the observed movement, which lie at the lowest level of the action hierarchy. In turn, this low-level kinematic inference supports higher-level inferences about deeper aspects of the observed actions: proximal goals and intentions. Finally, the inferred action representations can steer imitative responses, but interfere with the execution of different actions. Our simulations show that hierarchical active inference provides a unified account of action observation, understanding, learning and imitation and helps explain the neurobiological underpinnings of visuomotor cognition, including the multiple routes for action understanding in the dorsal and ventral streams and mirror mechanisms.
Acquired motor limits can be provoked by neurological lesions. Independently of the aetiologies, the lesions require patients to develop new coping strategies and adapt to the changed motor functionalities. In all of these occasions, what is defined as an assistive technology (AT) may represent a promising solution. The present work is a systematic review of the scientific AT-related literature published in the PubMed, Cinahl, and Psychinfo databases up to September 2022. This review was undertaken to summarise how the acceptance of AT is assessed in people with motor deficits due to neurological lesions. We review papers that (1) dealt with adults (≥18 years old) with motor deficits due to spinal cord or acquired brain injuries and (2) concerned user acceptance of hard AT. A total amount of 615 studies emerged, and 18 articles were reviewed according to the criteria. The constructs used to assess users’ acceptance mainly entail people’s satisfaction, ease of use, safety and comfort. Moreover, the acceptance constructs varied as a function of participants’ injury severity. Despite the heterogeneity, acceptability was mainly ascertained through pilot and usability studies in laboratory settings. Furthermore, ad-hoc questionnaires and qualitative methods were preferred to unstandardized protocols of measurement. This review highlights how people living with acquired motor limits greatly appreciate ATs. On the other hand, methodological heterogeneity indicates that evaluation protocols should be systematized and finely tuned.
A major issue in modern neuroscience is to understand how cell populations present multiple spatial and motor features during goal-directed movements. The direction and distance (depth) of arm movements often appear to be controlled independently during behavior, but it is unknown whether they share neural resources or not. Using information theory, singular value decomposition, and dimensionality reduction methods, we compare direction and depth effects and their convergence across three parietal areas during an arm movement task. All methods show a stronger direction effect during early movement preparation, whereas depth signals prevail during movement execution. Going from anterior to posterior sectors, we report an increased number of cells processing both signals and stronger depth effects. These findings suggest a serial direction and depth processing consistent with behavioral evidence and reveal a gradient of joint versus independent control of these features in parietal cortex that supports its role in sensorimotor transformations.
In the past, neuroscience was focused on individual neurons seen as the functional units of the nervous system, but this approach fell short over time to account for new experimental evidence, especially for what concerns associative and motor cortices. For this reason and thanks to great technological advances, a part of modern research has shifted the focus from the responses of single neurons to the activity of neural ensembles, now considered the real functional units of the system. However, on a microscale, individual neurons remain the computational components of these networks, thus the study of population dynamics cannot prescind from studying also individual neurons which represent their natural substrate. In this new framework, ideas such as the capability of single cells to encode a specific stimulus (neural selectivity) may become obsolete and need to be profoundly revised. One step in this direction was made by introducing the concept of “mixed selectivity,” the capacity of single cells to integrate multiple variables in a flexible way, allowing individual neurons to participate in different networks. In this review, the Authors outline the most important features of mixed selectivity and they also present recent works demonstrating its presence in the associative areas of the posterior parietal cortex. Finally, in discussing these findings, the Authors pose some open questions that could be addressed by future studies.
The dexterous control of our grasping actions relies on the cooperative activation of many brain areas. In the parietal lobe, two grasp-related areas collaborate to orchestrate an accurate grasping action: dorsolateral area AIP and dorsomedial area V6A. Single-cell recordings in monkeys and fMRI studies in humans have suggested that both these areas specify grip aperture and wrist orientation, but encode these grasping parameters differently, depending on the context. To elucidate the causal role of phAIP and hV6A, we stimulated these areas while participants were performing grasping actions (unperturbed grasping). rTMS over phAIP impaired the wrist orientation process, whereas stimulation over hV6A impaired grip aperture encoding. In a small percentage of trials, an unexpected reprogramming of grip aperture or wrist orientation was required (perturbed grasping). In these cases, rTMS over hV6A or over phAIP impaired reprogramming of both grip aperture and wrist orientation. These results represent the first direct demonstration of a different encoding of grasping parameters by two grasp-related parietal areas.
Despite the well-recognized role of the posterior parietal cortex (PPC) in processing sensory information to guide action, the differential encoding properties of this dynamic processing, as operated by different PPC brain areas, are scarcely known. Within the monkey’s PPC, the superior parietal lobule hosts areas V6A, PEc, and PE included in the dorso-medial visual stream that is specialized in planning and guiding reaching movements. In this study, a Convolutional Neural Network (CNN) approach is used to investigate how the information is processed in these areas. We trained two macaque monkeys to perform a delayed reaching task towards 9 positions (distributed on 3 different depth and direction levels) in the 3D peripersonal space. The activity of single cells was recorded from V6A, PEc, PE and fed to convolutional neural networks that were designed and trained to exploit the temporal structure of neuronal activation patterns, to decode the target positions reached by the monkey. Bayesian Optimization was used to define the main CNN hyper-parameters. In addition to discrete positions in space, we used the same network architecture to decode plausible reaching trajectories. We found that data from the most caudal V6A and PEc areas outperformed PE area in the spatial position decoding. The results support a dynamic encoding of the different phases and properties of the reaching movement differentially distributed over a network of interconnected areas. This study highlights the usefulness of neurons’ firing rate decoding via CNNs to improve our understanding of how sensorimotor information is encoded in PPC to perform reaching movements. The obtained results may have implications in the perspective of novel neuroprosthetic devices based on the decoding of these rich signals for faithfully carrying out patient’s intentions.
In daily life, we often are confronted with a combination of a visual search task and subsequent grasping of an object. Visual search research demonstrates strong evidence that the scene context affects where and how often we gaze within a given scene. In a grasping task, it is yet unknown whether different phases of object manipulation, such as reaching the object or transporting it from one location to another, are affected by the scene context. The present study investigated the effect of the scene context on human object manipulation in an interactive task in a realistic scenario. Specifically, pick-and-place task performance was evaluated while placing the objects of interest into typical everyday visual scenes. As the main result of the present study, the scene context had a significant effect on the search phase, but not on the transport and reach phases of the task. This strengthens the validity of transferring eye and hand movement knowledge in a grasping task performed in an isolated setting to a realistic scenario within a context-rich environment. The findings of this work provide insights into the potential development of supporting intention predicting systems.
Patients with lesions of the parieto-occipital cortex typically misreach visual targets that they correctly perceive (optic ataxia). Although optic ataxia was described more than 30 years ago, distinguishing this condition from physiological behavior using kinematic data is still far from being an achievement. In this study, combining kinematic analysis with machine learning methods, we compared the reaching performance of a patient with bilateral occipitoparietal damage with that of 10 healthy controls. Specifically, accurate predictions of the patient’s deviations were detected after the 20% of the movement execution in all the spatial positions tested. This classification based on initial trajectory decoding was possible for both directional and depth components of the movement, suggesting the possibility of applying this method to characterize pathological motor behavior.
Highlights
Parietal area V6A encodes sensory and motor activity in goal-directed arm movements
V6A acts as a state estimator of the arm position and movement
Gaze-dependent visual neurons and real-position cells encode action target in V6A
V6A uses the spotlight of attention to guide goal-directed movements of the hand
V6A hosts a priority map for the guidance of reaching arm movement
Acquired brain injury and spinal cord injury are leading causes of severe motor disabilities impacting a person’s autonomy and social life. Enhancing neurological recovery driven by neurogenesis and neuronal plasticity could represent future solutions; however, at present, recovery of activities employing assistive technologies integrating artificial intelligence is worthy of examining. MAIA (Multifunctional, adaptive, and interactive AI system for Acting in multiple contexts) is a human-centered AI aiming to allow end-users to control assistive devices naturally and efficiently by using continuous bidirectional exchanges among multiple sensorimotor information.
Aimed at exploring the acceptability of MAIA, semi-structured interviews (both individual interviews and focus groups) are used to prompt possible end-users (both patients and caregivers) to express their opinions about expected functionalities, outfits, and the services that MAIA should embed, once developed, to fit end-users needs.
End-user indications are expected to interest MAIA technical, health-related, and setting components. Moreover, psycho-social issues are expected to align with the technology acceptance model. In particular, they are likely to involve intrinsic motivational and extrinsic social aspects, aspects concerning the usefulness of the MAIA system, and the related ease to use. At last, we expect individual factors to impact MAIA: gender, fragility levels, psychological aspects involved in the mental representation of body image, personal endurance, and tolerance toward AT-related burden might be the aspects end-users rise in evaluating the MAIA project.
Perception and action are essential aspects in our day-to-day interactions with the environment. When we reach to grasp an object, the hand is pre-shaped according to the perceived intrinsic properties of the object, like size and shape. Within this context, years ago the dual-stream theory was proposed, suggesting independent streams for perception and action. However, nowadays the idea of a close action-perception coupling has gained momentum, suggesting that both processes (perception and action) are not entirely separate, but interact with each other, sharing a common representational code. The scientific literature on the effects of actions on perception is not very extensive. Some studies have reported how orientation tasks are influenced by the action plan, or even how the grasping context can alter size perception. These assumptions suggest that perception can be influenced by the action context, nevertheless, it remains to be known how motor actions and unpredictable size perturbations during action executions could influence perception.
In this study, we aimed to determine whether size perception and saccade amplitude before and after grasping movements were modified by horizontal size perturbations during movement execution and under the presence or absence of tactile feedback.
Two main discoveries were found. First, the horizontal size perception is modulated by the execution of grasping movements and, moreover, it relies on the presence of a horizontal size perturbation during the grasping action. Second, saccade responses exhibited amplitude modifications according to the types of target perturbations occurred in the previous action execution. These findings suggest that the combination of manual perceptual reports and corresponding saccade amplitudes are descriptive parameters of previous motor actions.
The superior parietal lobule (SPL) integrates somatosensory, motor, and visual signals to dynamically control arm movements. During reaching, visual and gaze signals are used to guide the hand to the desired target location, while proprioceptive signals allow to correct arm trajectory, and keep the limb in the final position at the end of the movement. Three SPL areas are particularly involved in this process: V6A, PEc, PE. Here, we evaluated the influence of eye and arm position on single neuron activity of these areas during the holding period at the end of arm reaching movements, when the arm is motionless and gaze and hand positions are aligned. Two male macaques (Macaca fascicularis) performed a foveal reaching task while single unit activity was recorded from areas V6A, PEc, and PE. We found that at the end of reaching movements the neurons of all these areas were modulated by both eye position and static position of the arm. V6A and PEc showed a prevalent combination of gaze and proprioceptive input, while PE seemed to encode these signals more independently. Our results demonstrate that all these SPL areas combine gaze and proprioceptive input to provide an accurate monitoring of arm movements.
Progressive addition lenses (PALs) are ophthalmic lenses to correct presbyopia by providing improvements of near and far vision in different areas of the lens, but distorting the periphery of the wearer's field of view. Distortion-related difficulties reported by PAL wearers include unnatural self-motion perception. Visual self-motion perception is guided by optic flow, the pattern of retinal motion produced by self-motion. We tested the influence of PAL distortions on optic flow-based heading estimation using a model of heading perception and a virtual reality-based psychophysical experiment. The model predicted changes of heading estimation along a vertical axis, depending on visual field size and gaze direction. Consistent with this prediction, participants experienced upwards deviations of self-motion when gaze through the periphery of the lens was simulated, but not for gaze through the center. We conclude that PALs may lead to illusions of self-motion which could be remedied by a careful gaze strategy.
Saccadic eye movements bring objects of interest onto our fovea. These gaze shifts are essential for visual perception of our environment and the interaction with the objects within it. They precede our actions and are thus modulated by current goals. It is assumed that saccadic adaptation, a recalibration process that restores saccade accuracy in case of error, is mainly based on an implicit comparison of expected and actual post-saccadic position of the target on the retina. However, there is increasing evidence that task demands modulate saccade adaptation and that errors in task performance may be sufficient to induce changes to saccade amplitude. We investigated if human participants are able to flexibly use different information sources within the post-saccadic visual feedback in task-dependent fashion. Using intra-saccadic manipulation of the visual input, participants were either presented with congruent post-saccadic information, indicating the saccade target unambiguously, or incongruent post-saccadic information, creating conflict between two possible target objects. Using different task instructions, we found that participants were able to modify their saccade behavior such that they achieved the goal of the task. They succeeded in decreasing saccade gain or maintaining it, depending on what was necessary for the task, irrespective of whether the post-saccadic feedback was congruent or incongruent. It appears that action intentions prime task-relevant feature dimensions and thereby facilitated the selection of the relevant information within the post-saccadic image. Thus, participants use post-saccadic feedback flexibly, depending on their intentions and pending actions.
The medial posterior parietal cortex (PPC) is involved in the complex processes of visuomotor integration. Its connections to the dorsal premotor cortex, which in turn is connected to the primary motor cortex (M1), complete the fronto-parietal network that supports important cognitive functions in the planning and execution of goal-oriented movements. In this study, we wanted to investigate the time-course of the functional connectivity at rest between the medial PPC and the M1 using dual-site transcranial magnetic stimulation in healthy humans. We stimulated the left M1 using a suprathreshold test stimulus to elicit motor-evoked potentials in the hand, and a subthreshold conditioning stimulus was applied over the left medial PPC at different inter-stimulus intervals (ISIs). The conditioning stimulus affected the M1 excitability depending on the ISI, with inhibition at longer ISIs (12 and 15 ms). We suggest that these modulations may reflect the activation of different parieto-frontal pathways, with long latency inhibitions likely recruiting polisynaptic pathways, presumably through anterolateral PPC.
In this paper, we trained multiple machine learning models to investigate if data from contemporary virtual reality hardware enables long- and short-term locomotion predictions - which is important for the MAIA project.
To create our data set, 18 participants walked through a virtual environment with different tasks. The recorded positional, orientation- and eye-tracking data was used to train a machine learning model predicting the future walking targets. Our results showed that the best model to predict the next time step was a Long short-term memory (LSTM) model using positional and orientation data with a mean error of 5.14 mm. The best model for predictions of 2.5 s was an LSTM using positional, orientation and eye-tracking data with a mean error of 65.73 cm. Gaze data offered the greatest predictive utility for long-term predictions of short distances. Our findings indicate that an LSTM model can be used to predict walking paths in VR. Moreover, our results suggest that eye-tracking data provides an advantage for this task.
Understanding the control of movement exerted by motor cortices is challenging and even more so for the associative areas of parietal cortex where many inputs are integrated. Moreover, while single neuron approaches can address how neural computations are carried out, we need to consider the entire neural population activity to get insight on what the brain is computing.
In our recent article entitled “Motor-like neural dynamics in two parietal areas during arm reaching”, published in the journal Progress in Neurobiology, using a population approach that modelled neural activity as a Hidden Markov process, we showed striking similarities between frontal and parietal cortex neural dynamics during arm movements in real, 3D space. Our findings are in agreement with recently proposed models of sensorimotor circuits and our approach can be further applied at the network level using simultaneous neural recordings from parietal and frontal areas.
In macaques, the superior parietal lobule (SPL) integrates information from various cortical sectors like visual (V) motor (M) and somatosensory (S) cortex and from the thalamus (TH) and thus forms an interface between perception and action. Recent functional imaging studies have identified putative human homologues of several macaque SPL areas. In our review article entitled "The superior parietal lobule of primates: a sensory-motor hub for interaction with the environment" published in the Special Issue: The sensory-cognitive Interplay: Insights into the neural mechanisms and circuits of the Journal of Integrative Neuroscience (2021, Vol.20 Issue (1): 157-171 DOI: 10.31083/j.jin.2021.01.334) we provide an account of the anatomical subdivisions, the connections and the function of the macaque SPL and highlight similarities with human SPL organization both under normal condition and after SPL lesions. Knowledge of this part of the brain could help understanding abnormal reaching and grasping behavior in humans and also fuel research on brain machine interfaces (BMI) enabling prosthetic arms and hand to perform more smooth and accurate actions In dynamic environments.
This review is particularly focused on the origin and type of visual information reaching the SPL, and on the functional role this information can play in guiding limb interaction with object in structured and dynamic environments. The knowledge of SPL in primates, considering that this structure shares anatomical and functional organization in human and non-human primates, could help in understanding how to guide human neuroprosthetic devices in complex natural contests.
