Solarflare’s creation of a ‘virtualization aware’ smart network adapter and modified Xen will allow direct, but safe, access to such adapters from guest operating systems in order to improve network performance.
Getting 10 Gb/s from Xen: Safe and FastDevice Access from Unprivileged Domains
Kieran Mansley, Greg Law, David Riddoch,Guido Barzini, Neil Turton, and Steven Pope
Solar?are Communications, Inc.{kmansley,glaw,driddoch,gbarzini,nturton,spope}@solarflare.com
Abstract. The networking performance available to Virtual Machines(VMs) can be low dueto theine?ciencies of transferring network packetsbetween the host domain and guests. This can limit the application-levelperformance of VMs on a 10Gb/s network. To improve network perfor-mance, we have created a "virtualization aware" smart network adapter1and modi?ed Xen to allow direct, but safe, access to such adapters fromguest operating systems. Networking overheadsare reducedconsiderably,and the host domain is removed as a bottleneck, resulting in signi?cantlyimproved performance.We describe our modi?cations to the Xen networking architecture thatallow guest kernels direct - but secure - access to the networking hard-ware, whilst preserving support for migration. We also describe brie?yhow the same technology is used to grant direct network access to user-level applications and thus provide even greater e?ciency in terms ofbandwidth, latency and CPU utilisation.
1 Introduction
Modern commodity servers are able to saturate 10Gb/s Ethernet net-works [1]. However, VM guests incur signi?cant additional overheadsdue to context-switching, data movement and passing packets betweenthe host domain and guests. The Xen [2] virtual machine architectureimposes relatively low overheads on networking compared with othervirtualization technologies, yet its performance is well below that of na-tive operating systems [3,4]. The privileged host VM quickly becomes abottleneck as all network tra?c must pass through it.This I/O bottleneck makes virtualization e?ectively impractical for cer-tain classes of application, including many HPC applications and highperformance servers. As 10Gb/s Ethernet moves more into the main-stream, the set of applications for which this is a barrier to adoption ofvirtualization will likely increase.This paper shows how allowing guest operating systems direct access tothe I/O hardware eliminates the bottleneck and software overheads usu-ally associated with virtualized I/O, and allows a virtualized guest toachieve performance comparable with a system running natively. Doing1 Xen is a trademark of XenSource, Inc. in the United States and other countries.so requires additional support from the hardware in order to multiplexthe device between multiple guests that may access it concurrently andalso to enforce isolation so that guests cannot gain privileges or compro-mise system integrity.The rest of this paper is structured as follows. Section 2 outlines thearchitecture for accelerated networking that we have added to Xen, whileSection 3 expands to describe our implementation and the results wehave obtained with a virtualization-aware network adapter. Section 4shows how the same techniques can be applied to user-level applications.Finally, Section 5 concludes.
2 Architecture
2.1 Xen Paravirtualized Network I/O
Paravirtualized network I/O in Xen is achieved through a pair of inter-linked drivers; netfront the "frontend driver" in the guest, and netbackthe "backend driver" in the host domain. The frontend and backendcommunicate through a region of shared memory and send each othervirtual interrupts using event channels. Together these form a channelthat supports the transfer of packets between host domain and guest.The upper edge of thefrontend driverpresents the interface of a standardnetwork devicedriver,allowing it tointerface tothe bottomof theguest'snetwork stack. The backend appears likewise and is usually con?guredto connect to a software bridge in the host OS. This allows it to com-municate with the host's network stack, other virtual machines' backenddrivers, and physical network interfaces and so the network beyond.Packets that arrive from a physical network interface are routed by thebridge to the appropriate backenddrivers, which in turn forward them tothe corresponding guests' frontend drivers. These then pass them on tothe network stack as if they had arrived directly at the guests. Packetssent by guests follow the same path in reverse. Packets sent from oneguest to another are routed between the corresponding backend driversby the bridge.
2.2 Acceleration Architecture
We have extended the ... [download for more]
