<?xml version="1.0" encoding="UTF-8"?>
<chapter id="ipc">
<?dbhtml filename="ipc.html"?>
<title>IPC</title>
<para>Due to the high intertask communication traffic, IPC becomes critical
subsystem for microkernels, putting high demands on the speed, latency and
reliability of IPC model and implementation. Although theoretically the use
of asynchronous messaging system looks promising, it is not often
implemented because of a problematic implementation of end user
applications. HelenOS implements a fully asynchronous messaging system but
with a special layer providing a user application developer a reasonably
synchronous multithreaded environment sufficient to develop complex
protocols.</para>
<section>
<title>Services provided by kernel</title>
<para>Every message consists of 4 numeric arguments (32-bit and 64-bit on
the corresponding platforms), from which the first one is considered a
method number on message receipt and a return value on answer receipt. The
received message contains identification of the incoming connection, so
that the receiving application can distinguish the messages between
different senders. Internally the message contains pointer to the
originating task and to the source of the communication channel. If the
message is forwarded, the originating task identifies the recipient of the
answer, the source channel identifies the connection in case of a hangup
response.</para>
<para>Every message must be eventually answered. The system keeps track of
all messages, so that it can answer them with appropriate error code
should one of the connection parties fail unexpectedly. To limit buffering
of the messages in the kernel, every process is has a limited account of
asynchronous messages it can send simultanously. If the limit is reached,
the kernel refuses to send any other message, until some active message is
answered.</para>
<para>To facilitate kernel-to-user communication, the IPC subsystem
provides notification messages. The applications can subscribe to a
notification channel and receive messages directed to this channel. Such
messages can be freely sent even from interrupt context as they are
primarily destined to deliver IRQ events to userspace device drivers.
These messages need not be answered, there is no party that could receive
such response.</para>
<section>
<title>Low level IPC</title>
<para>The whole IPC subsystem consists of one-way communication
channels. Each task has one associated message queue (answerbox). The
task can open connections (identified by phone id) to other tasks, send
and forward messages through these connections and answer received
messages. Every sent message is identified by a unique number, so that
the response can be later matched against it. The message is sent over
the phone to the target answerbox. Server application periodically
checks the answerbox and pulls messages from several queues associated
with it. After completing the requested action, server sends a reply
back to the answerbox of the originating task. If a need arises, it is
possible to <emphasis>forward</emphasis> a recevied message throught any
of the open phones to another task. This mechanism is used e.g. for
opening new connections.</para>
<para>The arguments contained in the message are completely arbitrary
and decided by the user. The low level part of kernel IPC fills in
appropriate error codes if there is an error during communication. It is
ensured that the applications are correctly notified about communication
state. If the outgoing connection is closed with the hangup message, the
target answerbox receives a hangup message. The connection
identification is not reused, until the hangup message is acknowledged
and all other pending messages are answered.</para>
<para>If the server side decides to hangup an incoming connection, it
does it by responding to any incoming message with an EHANGUP error
code. The connection is then immediately closed. The client connection
identification (phone id) is not reused, until the client issues hangup
system call to close the outgoing connection.</para>
<para>When a task dies (whether voluntarily or by being killes), cleanup
process is started which performs following tasks.</para>
<orderedlist>
<listitem>
<para>Hangs up all outgoing connections and sends hangup messages to
all target answerboxes.</para>
</listitem>
<listitem>
<para>Disconnects all incoming connections.</para>
</listitem>
<listitem>
<para>Disconnects from all notification channels.</para>
</listitem>
<listitem>
<para>Answers all unanswered messages from answerbox queues with
appropriate error code.</para>
</listitem>
<listitem>
<para>Waits until all outgoing messages are answered and all
remaining answerbox queues are empty.</para>
</listitem>
</orderedlist>
</section>
<section>
<title>System call IPC layer</title>
<para>On top of this simple protocol the kernel provides special
services closely related to the inter-process communication. A range of
method numbers is allocated and protocol is defined for these functions.
The messages are interpreted by the kernel layer and appropriate actions
are taken depending on the parameters of message and answer. </para>
<para>The kernel provides the following services:</para>
<itemizedlist>
<listitem>
<para>Creating new outgoing connection</para>
</listitem>
<listitem>
<para>Creating a callback connection</para>
</listitem>
<listitem>
<para>Sending an address space area</para>
</listitem>
<listitem>
<para>Asking for an address space area</para>
</listitem>
</itemizedlist>
</section>
</section>
<section>
<title>Userspace view</title>
<para>The conventional design of the asynchronous api seems to produce
applications with one event loop and several big switch statements.
However, by intensive utilization of user-space threads, it was possible
to create an environment that is not necesarilly restricted to this type
of event-driven programming and allows for more fluent expression of
application programs.</para>
<section>
<title>Single point of entry</title>
<para>Each tasks is associated with only one answerbox. If a
multi-threaded application needs to communicate, it must be not only
able to send a message, but it should be able to retrieve the answer as
well. If several threads pull messages from task answerbox, it is a
matter of fortune, which thread receives which message. If a particular
thread needs to wait for a message answer, an idle
<emphasis>manager</emphasis> task is found or a new one is created and
control is transfered to this manager task. The manager tasks pops
messages from the answerbox and puts them into appropriate queues of
running tasks. If a task waiting for a message is not running, the
control is transferred to it.</para>
<para>Very similar situation arises when a task decides to send a lot of
messages and reaches kernel limit of asynchronous messages. In such
situation 2 remedies are available - the userspace liberary can either
cache the message locally and resend the message when some answers
arrive, or it can block the thread and let it go on only after the
message is finally sent to the kernel layer. With one exception HelenOS
uses the second approach - when the kernel responds that maximum limit
of asynchronous messages was reached, control is transferred to manager
thread. The manager thread then handles incoming replies and when space
is available, sends the message to kernel and resumes application thread
execution.</para>
<para>If a kernel notification is received, the servicing procedure is
run in the context of the manager thread. Although it wouldn't be
impossible to allow recursive calling, it could potentially lead to an
explosion of manager threads. Thus, the kernel notification procedures
are not allowed to wait for a message result, they can only answer
messages and send new ones without waiting for their results. If the
kernel limit for outgoing messages is reached, the data is automatically
cached within the application. This behaviour is enforced automatically
and the decision making is hidden from developers view.</para>
</section>
<section>
<title>Synchronization problem</title>
<para>Unfortunately, in the real world is is never so easy. E.g. if a
server handles incoming requests and as a part of it's response sends
asynchronous messages, it can be easily prempted and other thread may
start intervening. This can happen even if the application utilizes only
1 kernel thread. Classical synchronization using semaphores is not
possible, as locking on them would block the thread completely and the
answer couldn't be ever processed. The IPC framework allows a developer
to specify, that the thread should not be preempted to any other thread
(except notification handlers) while still being able to queue messages
belonging to other threads and regain control when the answer
arrives.</para>
<para>This mechanism works transparently in multithreaded environment,
where classical locking mechanism (futexes) should be used. The IPC
framework ensures that there will always be enough free threads to
handle the threads requiring correct synchronization and allow the
application to run more user-space threads inside the kernel threads
without the danger of locking all kernel threads in futexes.</para>
</section>
<section>
<title>The interface</title>
<para></para>
</section>
</section>
</chapter>