Challenges
It is clear that sample-based techniques cannot offer a viable way of preserving or restoring many musical instruments. Furthermore, the aims of digital restoration would only be partially fulfilled by the mere storage of waveforms: if nothing about the instrument is known, the idea of the instrument as a document is irretrievably lost. The limitations imposed by sampling-based techniques can only be surpassed by adopting a radically different approach to understanding a musical instrument.
The grand challenge of NEMUS is thus not only to reproduce the sound of stringed historical instruments, but also to generate a wealth of knowledge about the mechanisms involved in sound production and transmission. Only upon a thorough understanding of the sounding object may one build a faithful, durable copy. NEMUS will therefore embrace, develop and expand cutting-edge research in physical modelling. The restoration of historical musical instruments by physical modelling presents unique challenges that have not been directly addressed before.
Challenge 1. Expanding focused acoustical studies in organology
Musical instruments have a rich history reflecting changes in technology, aesthetics, taste and society; understanding such changes is the essential goal of organology. There is a substantial amount of research from a historical and organological point of view, see e.g. [1-4] on the harpsichord. There are also a number of studies devoted to understanding single instruments, or single components of an instrument, from an acoustics perspective. The harpsichord was studied in [5,6,7]. The piano (particularly the piano soundboard) has received a lot of attention: aside from the afore mentioned work of Chabassier and associates, a number of authors have studied the piano analytically, numerically and experimentally [8-17]. Bowed instruments have also received quite a lot of attention, particularly the violin: a general review is given by Woodhouse [18], who has also written a number of other articles on many aspects of violin acoustics.
Studies addressing the issue of design changes from an acoustical perspective, however, are missing. An exception is represented by the recent projects by Chaigne on historic pianos of the Viennese tradition, as well as by the works of LeConte, Le Moyne and colleagues on the restoration of a rare harpsichord by Ioannes Couchet, aided by numerical investigation [19,20].
NEMUS will focus on the specific design of the 1679 Ioseph Ioannes Couchet single-manual harpsichord, a drawing of which you can see at the top of the page. This is one of the last dated instruments made in the Ruckers/Couchet family tradition. This instrument clearly belongs to the mainstream Ruckers/Couchet tradition established by Hans Ruckers (c.1533/5 - 1598) about 100 years earlier, but it also illustrates the evolution of the tradition with the incorporation of many features that became necessary because of later performance practice.NEMUS will simulate the acoustics of this instrument by physical modelling, and compare it to two classic harpsichords belonging to the Ruckers tradition : the 1638 Ioannes Ruckers double-manual harpsichord in the Russell Collection at the University of Edinburgh, and the 1646 double-manual harpsichord by Ruckers’ nephew Ioannes Couchet in the Museum of Musical Instruments in Brussels. This will offer a solid acoustics foundation to organological studies carried out by O’Brien [4,21-25]. NEMUS will hence bridge the gap between acoustics and organology.
Challenge 2. Generating cutting-edge techniques in physical modelling.
Musical instruments are very complex systems. In many cases, the vibration of certain parts must account for nonlinear effects. An example is offered by piano strings, where the phenomenon of “phantom partials” cannot be understood by linear theory alone. The plectrum-string, hammer-string and bow-string interactions are also nonlinear phenomena. At a numerical level, nonlinearities can only be treated by a specialised approach, within the context of mainstream numerical simulation. Finite difference and finite element techniques are indeed a most suitable choice for this purpose. Energy conservation in the discrete setting is a common feature across all these schemes. In turn, the schemes can be shown to preserve a numerical quantity at each time step, amenable to the energy of the continuous target system in the limit of infinite sample rate. This property yields stability of the associated numerical scheme [28].
However, the resulting schemes are often in the form of implicit nonlinear equations, which may only be solved using iterative methods which are computationally very costly. This has a negative impact on the overall performance of the schemes, particularly in regard to parallelisation and convergence [26].
NEMUS, however, aims at rendering the simulation of historical musical instruments under real-time, even for complex systems comprising various nonlinear lumped and distributed components. This is a very challenging task, and considerable analytical work is therefore needed to tackle it. Recently, a new class of schemes was investigated, with the intention of preserving the energy-preserving structure of the schemes, whilst at the same time avoiding expensive iterative procedures. Preliminary works have shown very promising results in the case of collision dynamics [27,28], as well as fully distributed nonlinearities in piano strings [29], with gains in the range of at least a factor of 10 in terms of computation time. The new schemes allow for the simulation of highly nonlinear systems under a unified, non-iterative framework. There are, however, a number of other issues that NEMUS will tackle. It is unclear whether the current algorithms may be further accelerated by exploiting the form of the associated linear update equation. NEMUS will also investigate the possibility of porting the algorithms to nonlinear modal equations, whose usage has been confined mostly to linear problems.
By eliminating the need for iterative solvers, efficient algorithms with unprecedented performance for nonlinear vibration will be generated: to date, distributed nonlinearities such as those encountered in a vibrating string cannot be resolved in real time. NEMUS aims at significantly improving these computational aspects by devising much faster schemes.
Challenge 3. Building advanced controllers with haptic feedback.
Many past and current projects in physical modelling have focused on the problem of control. At the heart of these investigations is the idea that an instrument can only be complete if there is some kind of interaction between the simulated instrument and the player. Today, this idea has found large support in the community; see, e.g., [30] and works on haptic feedback are many. This new branch of haptics has been called musical haptics [31]. Another branch of research is focusing on the integration of physical models in augmented and virtual realities; see the upcoming Marie-Curie ITN project VRACE as well as the works by Serafin and associates from the Multisensory Experience Lab at Aalborg University in Denmark [32-35] The recent Keytar project [36] is an example of the integration of a simple physical model with a virtual environment, coded in Unity3D.
An interesting trend of research has focused on the piano action. Starting from early theoretical works by Gillespie [37], andsubsequently taken up by Hirschkorn [38], today realistic feedback-controlled piano actions are a reality; see, e.g., [39]. However, these models rely on third-party multi-body solvers and do not take into account string vibration, and they have not been implemented along with a physical model of the full instrument.
The question of control is crucial for NEMUS: understanding how the player interfaced her/himself with a historical instrument could reveal a lot about the instrument’s musical scope and capacity. Modern instruments have seen, unsurprisingly, many modifications since their early inception. Many of these have little to do with the sound as such, and a lot to do with the player-instrument interaction: the evolution of the piano action is a key example here. By means of advanced physical models, NEMUS will uncover the processes involved in the player-instrument interaction. This is particularly relevant for keys (not just the piano), where the action mechanism has been subjected to various modifications over time. Unlike previous work, NEMUS will develop its own models and equation solvers, thus avoiding the need for third-party software. This will have several advantages: one is that of efficiency, as outlined earlier; another is the possibility of integrating the action mechanism in thephysical model of the complete instrument. Hence, part of NEMUS will be dedicated to building dedicated controllers with haptic feedback using the acquired knowledge. Originating from a unified framework, these models could then be much more easily incorporated in multimodal or virtual realities.
[1] A. G. Hess. The Transition from Harpsichord to Piano. Galpin Soc J, 6:75–94, 1953.
[2] F. Hubbard. Three Centuries of Harpsichord Making. University Press, Cambridge, Mass, USA, 1965.
[3] R. Russel. The harpsichord and the clavichord. Faber & Faber, London, UK, 1959.
[4] G. O’Brien. Ruckers: A harpsichord and virginal building tradition. Cambridge University Press, Cambridge, UK, 1990.
[5] V. Välimäki, H. Penttinen, J. Knif, M. Laurson, and C. Erkut. Sound synthesis of the harpsichord using a computationally efficient physical model. EURASIP J Adv Sig Pr, 2004(7):860718, 2004
[6] E. L. Kottick, K. D. Marshall, and T. J. Hendrickson. The acoustics of the harpsichord. Sci Am, 264(2):110–115, 1991.
[7] N. H. Fletcher. Analysis of the design and performance of harpsichords. Acustica, 37:139–147, 1977.
[8] K. Wogram. Acoustical Research on pianos: vibrational characteristics of the soundboard. Das Musikinstrument, 24:694702, 1980.
[9] H. Suzuki. Vibration and sound radiation of a piano soundboard. J Acoust Soc Am, 80(6):15731582, 1986.
[10] H. A. Conklin. Design and tone in the mechanoacoustic piano. Part II. Piano structure. J Acoust Soc Am, 100(2):695-708, 1996.
[11] N. Giordano. Simple model of a piano soundboard. J Acoust Soc Am, 102(2):1159-1168, 1997.
[12] N. Giordano. Sound production by a vibrating piano soundboard: Experiment. J Acoust Soc Am, 104(3):1648-1653, 1998.
[13] A. Chaigne, B. Cotte, and R. Viggiano. Dynamical properties of piano soundboards. J Acoust Soc Am, 133(4):2456–2466, 2013.
[14] K. Ege, X. Boutillon, and M. Rébillat. Vibroacoustics of the piano soundboard: (Non)linearity and modal properties in the low- and mid-frequency ranges. J Sound Vib, 332(5):1288–1305, 2013.
[15] X. Boutillon and K. Ege. Vibroacoustics of the piano soundboard: Reduced models, mobility synthesis, and acoustical radiation regime. J Sound Vib, 332(18):4261–4279, 2013.
[16] A. Mamou-Mani, J. Frelat, and C. Besnainou. Numerical simulation of a piano soundboard under downbearing. J Acoust Soc Am, 123(4):24012406, 2008.
[17] R. Corradi, S. Miccoli, G. Squicciarini, and P. Fazioli. Modal analysis of a grand piano soundboard at successive manufacturing stages. Appl Acoust, 125:113-127, 2017.
[18] J. Woodhouse. The acoustics of the violin: a review. Rep Prog Phys, 77:115901, 2014.
[19] S. Le Conte, S. Le Monye, F. Ollivier, and S. Vaiedelich. Using mechanical modelling and experimentation for the conservation of musical instruments. J Cult Herit, 13:161–164, 2012.
[20] S. Le Monye, S. Le Conte, F. Ollivier, F Frelat, J-C Battault, and S. Vaiedelich. Restoration of a 17th century harpsichord to playable condition: A numerical experimental study. J Acoust Soc Am, 131(1):888–896, 2012.
[21] G. O’Brien. Ioannes and Andreas Ruckers - a quatercentenary celebration. Early Music, 7(4):453–466, 1979.
[22] G. O’Brien. Some principles of eighteenth-century harpsichord stringing and their application. The Organ Yearbook, 12:160–176, 1981.
[23] G. O’Brien. The Double-Manual Harpsichord by Francis Coston, London, c.1725. Galpin Soc J, 47:1-42, 1994.
[24] G. O’Brien. The use of simple geometry and the local unit of measurement in the design of Italian stringed keyboard instruments: an aid to attribution and to organological analysis. Galpin Soc J, 52:108– 171, 1999.
[25] G. O’Brien. Criteria for the determination of original stringing in historical keyboard instruments: The cautionary tale of a 1785 Longman and Broderip harpsichord. In J. Koster, editor, The Historical Harpsichord, A Monograph Series in Honor of Frank Hubbard, Vol. 5: Aspects of Harpsichord Making in The British Isles, page 259278. Pendragon Press, Stuyvesant, USA, 2010.
[26] V. Chatziioannou, S. Schmutzhard, and S. Bilbao. On iterative solutions for numerical collision models. In Proceedings of the International Conference on Digital Audio Effects (DAFX-17), Edinburgh, UK, September 2017.
[27] M. Ducceschi and S. Bilbao. Non-iterative solvers for nonlinear problems: The case of collisions. In Proceedings of the International Conference on Digital Audio Effects (DAFX-19), Birmingham, UK, September 2019.
[28] S. Bilbao, M. Ducceschi, and C. Webb. A large-scale real-time modular physical modeling sound synthesis system. In Proceedings of the International Conference on Digital Audio Effects (DAFX-19), Birmingham, UK, September 2019.
[29] M. Ducceschi and S. Bilbao. Non-iterative, conservative schemes for geometrically exact nonlinear string vibration. In Proceedings of the International Congress On Acoustics (ICA 2019), Aachen, Ger- many, September 2016.
[30] G.W. Young, D. Murphy, and J. Weeter. A qualitative analysis of haptic feedback in music focused exercises. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME2017), Aalborg, Denmark, May 2017.
[31] S. Papetti and C. Saitis. Musical Haptics. Springer, Cham, Switzerland, 2018.
[32] S. Serafin, S. Trento, F. Grani, H. Perner-Wilson, S. Madgwick, and T. Mitchell. Controlling Physically Based Virtual Musical Instruments Using The Gloves. In Proceedings of New Interfaces for Musical Expression (NIME2014), London, UK, June 2014.
[33] S. Serafin, S. Dahl, R. Bresin, A. R. Jensenius, R. Unnpórsson, and V. Välimäki. Nordicsmc: A nordic university hub on sound and music computing. In Proceedings of the Sound and Music Computing Conference (SMC2018), Limassol, Cyprus, July 2018.
[34] S. Willemsen, Andersson N., S. Serafin, and S. Bilbao. Real-time control of large-scale modular physical models using the sensel morph. In Proceedings of the Sound and Music Computing Conference (SMC2019), Malaga, Spain, July 2019.
[35] J. Y. Tissieres, A. Vaseileiou, S. Zabetian, P. Delgado, S. Dahl, and S. Serafin. An expressive multi-dimensional physical modelling percussion instrument. In Proceedings of the Sound and Music Computing Conference (SMC2018), Limassol, Cyprus, July 2018.
[36] F. Fontana, A. Passalenti, and S. Serafin. Keytar: melodic control of multisensory feedback from virtual strings. In Proceedings of the International Conference on Digital Audio Effects (DAFX-19), Birmingham, UK, September 2019.
[37] B. Gillespie and M. Cutkosky. Dynamical modeling of the grand piano action. In Proceedings of the International Computer Music Conference (ICMC1992), San Jose, USA, 1992.
[38] M. Hirschkorn. Dynamic model of a piano action mechanism. Master’s thesis, University of Waterloo, 2004.
[39] S. Timmermans, A.-E. Ceulemans, and P. Fisette. A haptic piano keyboard based on a real-time multi- body model of the action. In Proceedings of the 26th International Congress on Sound and Vibration (ICSV26), Montreal, Canada, 2019.