RP1M: Robot Piano-Playing 1 Million Dataset

Although fingering is highly personalized, generally speaking, it helps pianists to press keys timely and efficiently. Motivated by this, apart from maximizing the key pressing rewards, we also aim to minimize the moving distances of fingers. Specifically, at time step \(t\), for the \(i\)-th key \(k^i\) to press, we use the \(j\)-th finger \(f^j\) to press this key such that the overall moving cost is minimized. We define the minimized cumulative moving distance as \(d_t^{\text{OT}} \in \mathbb{R}^+\), which is given by:

\[ \begin{split} d_t^{\text{OT}} =& \min_{w_t} \sum_{(i,j)\in K_t \times F}w_t(k^i, f^j)\cdot c_t(k^i, f^j), ~~ \text{s.t.}, ~~ i)~ \sum_{j\in F}w_t(k^i, f^j) = 1, ~~ \text{for} ~~ i \in K_t, \\ & ii)~ \sum_{i\in K_t}w_t(k^i, f^j) \leq 1, ~~ \text{for} ~~ j \in F, ~~~ iii)~~ w_t(k^i, f^j) \in \{0, 1 \}, ~~ \text{for} ~~ (i, j) \in K_t \times F. \end{split} \]

Here, \(K_t\) represents the set of keys to press at time step \(t\) and \(F\) represents the fingers of the robot hands. \(c_t(k^i, f^j)\) represents the cost of moving finger \(f^j\) to piano key \(k^i\) at time step \(t\) calculated by their Euclidean distance. \(w_t(k^i, f^j)\) is a boolean weight. In our case, it enforces that each key in \(K_t\) will be pressed by only one finger in \(F\), and each finger presses at most one key. The constrained optimization problem in the equation is an optimal transport problem. Intuitively, it tries to find the best "transport" strategy such that the overall cost of moving (a subset of) fingers \(F\) to keys \(K_t\) is minimized.

We compare the fingering discovered by the agent itself and the human annotations. We visualize the sample trajectory of playing French Suite No.5 Sarabande and the corresponding fingering. The agent discovers different fingering compared to humans. For example, for the right hand, humans mainly use the middle and ring fingers, while our agent uses the thumb and first finger. Furthermore, in some cases, human annotations are not suitable for the robot hand due to different morphologies. For example, in the second time step of the figure, the human uses the first finger and ring finger. However, due to the mechanical limitation of the robot hand, it can not press keys that far apart with these two fingers, thus mimicking human fingering will miss one key. Instead, our agent discovered to use the thumb and little finger, which satisfies the hardware limitation and accurately presses the target keys.