\section{The Vision}

XORP stands for eXtensible Open Router Platform.  It is called an {\it
extensible} router platform for good reason.  We believe that only by
designing for extensibility from the outset can we simultaneously
satisfy several different user groups:
\begin{itemize}
\vspace{-0.07in}
\item Network researchers needing a platform for experimentation.
\vspace{-0.07in}
\item Network operators needing a low-cost stable routing platform on
commodity hardware.
\vspace{-0.07in}
\item Network equipment vendors with special purpose hardware.
\vspace{-0.07in}
\item Network application writers looking for an open platform to
support their applications.
\end{itemize}
\vspace{-0.05in}
Only time will tell if we're right, but in this paper we'll try and
explain where we hope to go from here.  Of course we'll only get there
with good luck and a lot of help from all of these communities.  First
we'll discuss the XORP architecture, then return to how it helps these
diverse groups.

\subsection{XORP Overview}

XORP divides into two subsystems. The user-level subsystem consists of the routing protocols themselves, along with routing
information bases and support processes. The kernel-level subsystem manages the forwarding path and provides APIs for the user-level to access. The goal is for almost all of the user-level code to be agnostic as to the details of the forwarding path.

User-level XORP has a multi-process architecture, with
one process per routing protocol, plus extra processes for management,
configuration, and coordination.  To enable extensibility we designed
a novel inter-process communication mechanism, XORP Resource
Locators (XRLs), for communication between these modules.  

%The lower level uses the Click modular router~\cite{click}, a
%modular, extensible toolkit for packet processing on conventional PCs.
%Further work will help this architecture span a
%large range of hardware forwarding platforms, from commodity
%PC hardware, through mid-range PC-based platforms
%enhanced with intelligent network interfaces (such as Intel's
%IXP1200~\cite{ixp1200,scoutixp}), to high-end hardware-based forwarding engines.
%We may also support alternative forwarding paths, such as the
%FreeBSD forwarding path with AltQ queuing extensions~\cite{altq}
%or an alternative extensible forwarding path such as
%Scout~\cite{pathrouter}.
%Some forwarding path choices may influence the functionality available
%for end users to extend or change.  But for many aspects of research, such
%as routing protocol experiments that don't
%require access to the forwarding path, a conventional FreeBSD
%lower level would suffice.

\begin{floatingfigure}[l]{3.0in} %[h]
\centerline{\psfig{figure=processes3.ps,width=3.0in}}
\caption{XORP High-level Processes}
\label{fig:processes}
\vspace{-0.1in}
\end{floatingfigure}

Figure~\ref{fig:processes} shows a XORP router's user-level processes
and forwarding engine.
The shared user-level processes are the XORP architecture's most innovative
feature. Four core processes are particularly worthy of comment.

The \emph{router manager}
process manages the router as a whole.  It maintains
configuration information; starts processes such as routing
protocols, as required by the configuration; and restarts failed
processes as necessary.

The \emph{ipc finder} process stores mappings between application
requests, such as ``What are this router's interfaces?'', and the
particular IPC calls necessary to answer those requests.
Think of it as an IPC redirector:
when an application wishes to make an IPC
call, it consults the finder to discover how to do it.  Thus it is easy to change
how application requests are handled at run-time.  We
can also trace XORP's internal communications by asking the finder to map
abstract requests to sequences of IPCs, some for tracing and some for
doing the work.  

%% XXX This is duptext from Eddie and Orion editing at the same time.
%% Either this or the next paragraph should go.

%The \emph{forwarding engine abstraction} process, or FEA, abstracts the
%details of how the router's forwarding path is implemented.
%Routing protocols talk to the FEA to install routes and discover properties
%of interfaces, for example. 
%Interfaces to the FEA remain the same regardless of whether the forwarding
%path is implemented in Click, in a conventional kernel, or even in external
%hardware.
%Again, XORP processes can bypass the FEA when required.

The \emph{routing information base} process (RIB) receives routes from
the routing processes, and arbitrates which routes should be
propagated into the forwarding path, or redistributed to other routing
processes.  The forwarding path is managed
by the \emph{forwarding engine abstraction} process (FEA).  The FEA
abstracts the details of how the forwarding path of the router is
implemented and as a result, the routing processes are agnostic to
whether the forwarding plane is Click \cite{click} based, a conventional UNIX
kernel, or novel forwarding hardware  The FEA manages the networking
interfaces and forwarding table in the router, and provides information to routing processes
about the interface properties and events occurring on interfaces, such as an
interface being taken down.


\subsection{Extensibility}

XORP's design encourages the construction of useful
interfaces through multi-process design.
A routing protocol process, for example, must communicate with other
processes to install routes and discover information about the router
itself.
Open inter-process interfaces, built in to the system from the beginning,
form the basic source of user-level XORP's extensibility.

%
\begin{floatingfigure}[r]{3.3in} %[h]
%\vspace{-0.05in}
\centerline{\psfig{figure=xrl.ps,width=3.2in}}
\caption{An example XRL}
\label{fig:xrl}
\vspace{-0.1in}
\end{floatingfigure}

\def\xrl#1{\textsf{\small #1}} Most inter-process communication within
XORP takes place via XRLs.  XRLs resemble the Web's URLs. They specify
in human-readable form the type of IPC transport desired (the
``protocol''), the abstract name for the entity being communicated
with, the method being invoked, and a list of named arguments.  Unlike
URLs, they can also specify the nature of the response expected.
Figure \ref{fig:xrl} shows one of the XRLs for
the forwarding engine abstraction (FEA) process in
human readable form. The initial `\xrl{finder:}' tag specifies the
protocol; in this case the actual protocol has not yet been resolved.
The first time this XRL is called, the client contacts the
finder, which redirects to a new XRL containing the
actual protocol and parameters to be used to
contact the current FEA.  Subsequent communication bypasses the finder.

%The XRL library, which all of our processes link against, takes an XRL
%and performs argument marshaling, then it invokes the specified transport
%mechanism, and handles the response or any
%failure that might occur.  Unlike many RPC or remote method invocation
%mechanisms, XRLs don't try and hide from the programmer that
%off-process communication is taking place.  While this makes the
%programmer's job harder at first, it is essential for robustness that
%the programmer is made aware that the failure modes of IPC are different from those of
%a local procedure call.  
%To help the programmer and improve performance, an IDL and a stub
%generator exist, so most XRL clients never need to parse the human
%readable form.

One motivation for XRLs was to encapsulate existing
protocols within our consistent IPC framework.   For example, we
might wish to run third-party software that uses SNMPv3 for
configuration. 
To integrate this software into our XRL-based management framework, we
might write an SNMP `protocol family' for the XRL client library.
Then XORP processes could transparently interoperate with the third-party
software via XRLs that start with `\xrl{snmp:}'.
%XRLs are general enough to encompass simple
%communication with the kernel via \texttt{ioctl}s, and even signaling
%via \texttt{kill()}.  At the present time, we have not had a need to
%add this functionality, but should the need arise, our architecture
%would support it.  The current XRL library supports
%XORP-specific protocols for remote procedure call, one layered over
%TCP and the other over UDP, and a local procedure call mechanism for
%intra-process communication.

\subsubsection{XRL Example: Command-line Interface}

One of the biggest issues faced by an {extensible} router is 
the integration of separately maintained components into a coherent
system.  
%Consider the interaction between management mechanisms such
%as a command-line interface (CLI) and a new routing protocol.  The
%author of each of the management processes has no knowledge of future
%routing protocols.  At the same time, the author of each routing
%protocol has no knowledge of future management mechanisms. 
Our solution is for all management, including initial configuration, to
take place using XRLs.  To add support for a specific management
mechanism, such as SNMP or a command-line interface, the protocol
implementor writes simple text files that map management requests to
XRL calls.  These thin mapping files are easy enough to write that
third parties might add them as new management interfaces become
available.

To get more concrete, our configuration manager has a strict hierarchy
of configuration parameters, which is directly reflected in our
command line interface (CLI).  The configuration manager takes a
directory of template files, which define the possible configurable
parameters for each XORP routing protocol, and generates mappings of
configuration parameters to XRL dispatches.  The designer of a new
routing protocol can simply supply a template file specifying the new
functionality provided.  The configuration manager can read the
template file, discover the new functionality, and know how to
communicate with the process.  The new functionality is then
immediately available through the CLI.
For example, a fragment of a router configuration file, and the
corresponding part of a template file might look like:
%\vspace{-0.1in}

\noindent\framebox[\textwidth][l]{
\begin{minipage}{6in}
\begin{multicols}{2}
\noindent \sf \small protocols ospf \{\\
\parindent 10pt 
\indent     router-id: 128.16.64.1\\
\indent     area 128.16.0.1 \{\\\
\parindent 20pt 
\indent       interface xl0 \{\\
\parindent 30pt 
\indent         hello-interval: 30\\
\parindent 20pt 
\vspace{-0.1in}
\indent       \}\\
\parindent 10pt 
\vspace{-0.1in}
\indent     \}\\
\}
\eject
\noindent \sf \small hello-interval: uint \{ \\
\parindent 10pt 
\indent \%set: xrl ``ospf/ospf/1.0/set\_hello\_interval?\\
\parindent 60pt 
\indent if:txt=\$(IFNAME)\&interval:i32=\$(VALUE)'';\\
\parindent 10pt 
\indent \%get: xrl ``ospf/ospf/1.0/hello\_interval?if:txt\\
\parindent 60pt 
\indent -$>$ interval:i32'';\\
\}
\end{multicols}
\end{minipage}
}

\subsection{Robustness}

The routing and coordination processes in XORP run in user space on a
regular UNIX operating system.  Routing processes are protected
from each other and can have their resources constrained according to
administrative preference.  Furthermore, routing processes can crash
without affecting the kernel, forwarding plane, or each other.  And if
a routing protocol does crash, the RIB will remove its routes
from the forwarding engine, and optionally inform the re-starting
routing process of the routes it previously held.

%Multiple processes are used in Zebra~\cite{zebra} and Cisco's proposed ENA
%router operating system, but not by some of the larger
%commercial vendors today.

A significant aspect of robustness is security.  One benefit of
being forwarding-path agnostic is that we can abstract
privileged operations, such as sending on a raw socket, into
the FEA via XRLs.  This allows us to run many routing protocols
in a sandbox. They have no interaction with the outside world except
through XRLs and packets, and so a vulnerability in a routing
protocol is far more difficult to escalate into full control of the
router.  Untrusted code might even be run in a virtual
machine such as provided by Xen\cite{xen}.

%Robustness in the forwarding path is also important, but solutions
%such as memory protection that work well in user-space are not
%acceptable.  In the Click forwarding path robustness comes from the
%granularity and simplicity of Click's components.  Each element is
%small enough to be well understood and tested in isolation.  And since
%many of Click's components can be run and debugged in user-space,
%confidence about their robustness can be attained before being used in
%the kernel.

