Subversion Repositories HelenOS-doc

<?xml version="1.0" encoding="UTF-8"?>

<chapter id="mm">

  <?dbhtml filename="mm.html"?>

  <title>Memory management</title>

  <para>In previous chapters, this book described the scheduling subsystem as

  the creator of the impression that threads execute in parallel. The memory

  management subsystem, on the other hand, creates the impression that there

  is enough physical memory for the kernel and that userspace tasks have the

  entire address space only for themselves.</para>

  <section>

    <title>Physical memory management</title>

    <section id="zones_and_frames">

      <title>Zones and frames</title>

      <para>HelenOS represents continuous areas of physical memory in

      structures called frame zones (abbreviated as zones). Each zone contains

      information about the number of allocated and unallocated physical

      memory frames as well as the physical base address of the zone and

      number of frames contained in it. A zone also contains an array of frame

      structures describing each frame of the zone and, in the last, but not

      the least important, front, each zone is equipped with a buddy system

      that faciliates effective allocation of power-of-two sized block of

      frames.</para>

      <para>This organization of physical memory provides good preconditions

      for hot-plugging of more zones. There is also one currently unused zone

      attribute: <code>flags</code>. The attribute could be used to give a

      special meaning to some zones in the future.</para>

      <para>The zones are linked in a doubly-linked list. This might seem a

      bit ineffective because the zone list is walked everytime a frame is

      allocated or deallocated. However, this does not represent a significant

      performance problem as it is expected that the number of zones will be

      rather low. Moreover, most architectures merge all zones into

      one.</para>

      <para>For each physical memory frame found in a zone, there is a frame

      structure that contains number of references and data used by buddy

      system.</para>

    </section>

    <section id="frame_allocator">

      <title>Frame allocator</title>

      <para>The frame allocator satisfies kernel requests to allocate

      power-of-two sized blocks of physical memory. Because of zonal

      organization of physical memory, the frame allocator is always working

      within a context of some frame zone. In order to carry out the

      allocation requests, the frame allocator is tightly integrated with the

      buddy system belonging to the zone. The frame allocator is also

      responsible for updating information about the number of free and busy

      frames in the zone. <figure>

          <mediaobject id="frame_alloc">

            <imageobject role="html">

              <imagedata fileref="images/frame_alloc.png" format="PNG" />

            </imageobject>

            <imageobject role="fop">

              <imagedata fileref="images.vector/frame_alloc.svg" format="SVG" />

            </imageobject>

          </mediaobject>

          <title>Frame allocator scheme.</title>

        </figure></para>

      <formalpara>

        <title>Allocation / deallocation</title>

        <para>Upon allocation request via function <code>frame_alloc</code>,

        the frame allocator first tries to find a zone that can satisfy the

        request (i.e. has the required amount of free frames). Once a suitable

        zone is found, the frame allocator uses the buddy allocator on the

        zone's buddy system to perform the allocation. During deallocation,

        which is triggered by a call to <code>frame_free</code>, the frame

        allocator looks up the respective zone that contains the frame being

        deallocated. Afterwards, it calls the buddy allocator again, this time

        to take care of deallocation within the zone's buddy system.</para>

      </formalpara>

    </section>

    <section id="buddy_allocator">

      <title>Buddy allocator</title>

      <para>In the buddy system, the memory is broken down into power-of-two

      sized naturally aligned blocks. These blocks are organized in an array

      of lists, in which the list with index i contains all unallocated blocks

      of size <mathphrase>2<superscript>i</superscript></mathphrase>. The

      index i is called the order of block. Should there be two adjacent

      equally sized blocks in the list i<mathphrase />(i.e. buddies), the

      buddy allocator would coalesce them and put the resulting block in list

      <mathphrase>i + 1</mathphrase>, provided that the resulting block would

      be naturally aligned. Similarily, when the allocator is asked to

      allocate a block of size

      <mathphrase>2<superscript>i</superscript></mathphrase>, it first tries

      to satisfy the request from the list with index i. If the request cannot

      be satisfied (i.e. the list i is empty), the buddy allocator will try to

      allocate and split a larger block from the list with index i + 1. Both

      of these algorithms are recursive. The recursion ends either when there

      are no blocks to coalesce in the former case or when there are no blocks

      that can be split in the latter case.</para>

      <para>This approach greatly reduces external fragmentation of memory and

      helps in allocating bigger continuous blocks of memory aligned to their

      size. On the other hand, the buddy allocator suffers increased internal

      fragmentation of memory and is not suitable for general kernel

      allocations. This purpose is better addressed by the <link

      linkend="slab">slab allocator</link>.<figure>

          <mediaobject id="buddy_alloc">

            <imageobject role="html">

              <imagedata fileref="images/buddy_alloc.png" format="PNG" />

            </imageobject>

            <imageobject role="fop">

              <imagedata fileref="images.vector/buddy_alloc.svg" format="SVG" />

            </imageobject>

          </mediaobject>

          <title>Buddy system scheme.</title>

        </figure></para>

      <section>

        <title>Implementation</title>

        <para>The buddy allocator is, in fact, an abstract framework wich can

        be easily specialized to serve one particular task. It knows nothing

        about the nature of memory it helps to allocate. In order to beat the

        lack of this knowledge, the buddy allocator exports an interface that

        each of its clients is required to implement. When supplied with an

        implementation of this interface, the buddy allocator can use

        specialized external functions to find a buddy for a block, split and

        coalesce blocks, manipulate block order and mark blocks busy or

        available.</para>

        <formalpara>

          <title>Data organization</title>

          <para>Each entity allocable by the buddy allocator is required to

          contain space for storing block order number and a link variable

          used to interconnect blocks within the same order.</para>

          <para>Whatever entities are allocated by the buddy allocator, the

          first entity within a block is used to represent the entire block.

          The first entity keeps the order of the whole block. Other entities

          within the block are assigned the magic value

          <constant>BUDDY_INNER_BLOCK</constant>. This is especially important

          for effective identification of buddies in a one-dimensional array

          because the entity that represents a potential buddy cannot be

          associated with <constant>BUDDY_INNER_BLOCK</constant> (i.e. if it

          is associated with <constant>BUDDY_INNER_BLOCK</constant> then it is

          not a buddy).</para>

        </formalpara>

      </section>

    </section>

    <section id="slab">

      <title>Slab allocator</title>

      <para>The majority of memory allocation requests in the kernel is for

      small, frequently used data structures. The basic idea behind the slab

      allocator is that commonly used objects are preallocated in continuous

      areas of physical memory called slabs<footnote>

          <para>Slabs are in fact blocks of physical memory frames allocated

          from the frame allocator.</para>

        </footnote>. Whenever an object is to be allocated, the slab allocator

      returns the first available item from a suitable slab corresponding to

      the object type<footnote>

          <para>The mechanism is rather more complicated, see the next

          paragraph.</para>

        </footnote>. Due to the fact that the sizes of the requested and

      allocated object match, the slab allocator significantly reduces

      internal fragmentation.</para>

      <para>Slabs of one object type are organized in a structure called slab

      cache. There are ususally more slabs in the slab cache, depending on

      previous allocations. If the the slab cache runs out of available slabs,

      new slabs are allocated. In order to exploit parallelism and to avoid

      locking of shared spinlocks, slab caches can have variants of

      processor-private slabs called magazines. On each processor, there is a

      two-magazine cache. Full magazines that are not part of any

      per-processor magazine cache are stored in a global list of full

      magazines.</para>

      <para>Each object begins its life in a slab. When it is allocated from

      there, the slab allocator calls a constructor that is registered in the

      respective slab cache. The constructor initializes and brings the object

      into a known state. The object is then used by the user. When the user

      later frees the object, the slab allocator puts it into a processor

      private magazine cache, from where it can be precedently allocated

      again. Note that allocations satisfied from a magazine are already

      initialized by the constructor. When both of the processor cached

      magazines get full, the allocator will move one of the magazines to the

      list of full magazines. Similarily, when allocating from an empty

      processor magazine cache, the kernel will reload only one magazine from

      the list of full magazines. In other words, the slab allocator tries to

      keep the processor magazine cache only half-full in order to prevent

      thrashing when allocations and deallocations interleave on magazine

      boundaries. The advantage of this setup is that during most of the

      allocations, no global spinlock needs to be held.</para>

      <para>Should HelenOS run short of memory, it would start deallocating

      objects from magazines, calling slab cache destructor on them and

      putting them back into slabs. When a slab contanins no allocated object,

      it is immediately freed.</para>

      <para><figure>

          <mediaobject id="slab_alloc">

            <imageobject role="html">

              <imagedata fileref="images/slab_alloc.png" format="PNG" />

            </imageobject>

          </mediaobject>

          <title>Slab allocator scheme.</title>

        </figure></para>

      <section>

        <title>Implementation</title>

        <para>The slab allocator is closely modelled after OpenSolaris slab

        allocator by Jeff Bonwick and Jonathan Adams with the following

        exceptions:<itemizedlist>

            <listitem>

               empty slabs are immediately deallocated and

            </listitem>

            <listitem>

              <para>empty magazines are deallocated when not needed.</para>

            </listitem>

          </itemizedlist>The following features are not currently supported

        but would be easy to do: <itemizedlist>

            <listitem>

               cache coloring and

            </listitem>

            <listitem>

               dynamic magazine grow (different magazine sizes are already supported, but the allocation strategy would need to be adjusted).

            </listitem>

          </itemizedlist></para>

        <section>

          <title>Allocation/deallocation</title>

          <para>The following two paragraphs summarize and complete the

          description of the slab allocator operation (i.e.

          <code>slab_alloc</code> and <code>slab_free</code>

          operations).</para>

          <formalpara>

            <title>Allocation</title>

            <para><emphasis>Step 1.</emphasis> When an allocation request

            comes, the slab allocator checks availability of memory in the

            current magazine of the local processor magazine cache. If the

            available memory is there, the allocator just pops the magazine

            and returns pointer to the object.</para>

            <para><emphasis>Step 2.</emphasis> If the current magazine in the

            processor magazine cache is empty, the allocator will attempt to

            swap it with the last magazine from the cache and return to the

            first step. If also the last magazine is empty, the algorithm will

            fall through to Step 3.</para>

            <para><emphasis>Step 3.</emphasis> Now the allocator is in the

            situation when both magazines in the processor magazine cache are

            empty. The allocator reloads one magazine from the shared list of

            full magazines. If the reload is successful (i.e. there are full

            magazines in the list), the algorithm continues with Step

            1.</para>

            <para><emphasis>Step 4.</emphasis> In this fail-safe step, an

            object is allocated from the conventional slab layer and a pointer

            to it is returned. If also the last magazine is full,</para>

          </formalpara>

          <formalpara>

            <title>Deallocation</title>

            <para><emphasis>Step 1.</emphasis> During a deallocation request,

            the slab allocator checks if the current magazine of the local

            processor magazine cache is not full. If yes, the pointer to the

            objects is just pushed into the magazine and the algorithm

            returns.</para>

            <para><emphasis>Step 2.</emphasis> If the current magazine is

            full, the allocator will attempt to swap it with the last magazine

            from the cache and return to the first step. If also the last

            magazine is empty, the algorithm will fall through to Step

            3.</para>

            <para><emphasis>Step 3.</emphasis> Now the allocator is in the

            situation when both magazines in the processor magazine cache are

            full. The allocator moves one magazine to the shared list of full

            magazines. The algoritm continues with Step 1.</para>

          </formalpara>

        </section>

      </section>

    </section>

  </section>

  <section>

    <title>Virtual memory management</title>

    <section>

      <title>Introduction</title>

      <para>Virtual memory is a special memory management technique, used by

      kernel to achieve a bunch of mission critical goals. <itemizedlist>

          <listitem>

             Isolate each task from other tasks that are running on the system at the same time.

          </listitem>

          <listitem>

             Allow to allocate more memory, than is actual physical memory size of the machine.

          </listitem>

          <listitem>

             Allowing, in general, to load and execute two programs that are linked on the same address without complicated relocations.

          </listitem>

        </itemizedlist></para>

      <para><!--

                TLB shootdown ASID/ASID:PAGE/ALL.

                TLB shootdown requests can come in asynchroniously

                so there is a cache of TLB shootdown requests. Upon cache overflow TLB shootdown ALL is executed

                <para>

                        Address spaces. Address space area (B+ tree). Only for uspace. Set of syscalls (shrink/extend etc).

                        Special address space area type - device - prohibits shrink/extend syscalls to call on it.

                        Address space has link to mapping tables (hierarchical - per Address space, hash - global tables).

                </para>

--></para>

    </section>

    <section>

      <title>Paging</title>

      <para>Virtual memory is usually using paged memory model, where virtual

      memory address space is divided into the <emphasis>pages</emphasis>

      (usually having size 4096 bytes) and physical memory is divided into the

      frames (same sized as a page, of course). Each page may be mapped to

      some frame and then, upon memory access to the virtual address, CPU

      performs <emphasis>address translation</emphasis> during the instruction

      execution. Non-existing mapping generates page fault exception, calling

      kernel exception handler, thus allowing kernel to manipulate rules of

      memory access. Information for pages mapping is stored by kernel in the

      <link linkend="page_tables">page tables</link></para>

      <para>The majority of the architectures use multi-level page tables,

      which means need to access physical memory several times before getting

      physical address. This fact would make serios performance overhead in

      virtual memory management. To avoid this <link linkend="tlb">Traslation

      Lookaside Buffer (TLB)</link> is used.</para>

    </section>

    <section>

      <title>Address spaces</title>

      <section>

        <title>Address space areas</title>

        <para>Each address space consists of mutually disjunctive continuous

        address space areas. Address space area is precisely defined by its

        base address and the number of frames/pages is contains.</para>

        <para>Address space area , that define behaviour and permissions on

        the particular area. <itemizedlist>

            <listitem>

              <emphasis>AS_AREA_READ</emphasis>

               flag indicates reading permission.

            </listitem>

            <listitem>

              <emphasis>AS_AREA_WRITE</emphasis>

               flag indicates writing permission.

            </listitem>

            <listitem>

              <emphasis>AS_AREA_EXEC</emphasis>

               flag indicates code execution permission. Some architectures do not support execution persmission restriction. In this case this flag has no effect.

            </listitem>

            <listitem>

              <emphasis>AS_AREA_DEVICE</emphasis>

               marks area as mapped to the device memory.

            </listitem>

          </itemizedlist></para>

        <para>Kernel provides possibility tasks create/expand/shrink/share its

        address space via the set of syscalls.</para>

      </section>

      <section>

        <title>Address Space ID (ASID)</title>

        <para>When switching to the different task, kernel also require to

        switch mappings to the different address space. In case TLB cannot

        distinguish address space mappings, all mapping information in TLB

        from the old address space must be flushed, which can create certain

        uncessary overhead during the task switching. To avoid this, some

        architectures have capability to segregate different address spaces on

        hardware level introducing the address space identifier as a part of

        TLB record, telling the virtual address space translation unit to

        which address space this record is applicable.</para>

        <para>HelenOS kernel can take advantage of this hardware supported

        identifier by having an ASID abstraction which is somehow related to

        the corresponding architecture identifier. I.e. on ia64 kernel ASID is

        derived from RID (region identifier) and on the mips32 kernel ASID is

        actually the hardware identifier. As expected, this ASID information

        record is the part of <emphasis>as_t</emphasis> structure.</para>

        <para>Due to the hardware limitations, hardware ASID has limited

        length from 8 bits on ia64 to 24 bits on mips32, which makes it

        impossible to use it as unique address space identifier for all tasks

        running in the system. In such situations special ASID stealing

        algoritm is used, which takes ASID from inactive task and assigns it

        to the active task.</para>

        <para><classname>ASID stealing algoritm here.</classname></para>

      </section>

    </section>

    <section>

      <title>Virtual address translation</title>

      <section id="page_tables">

        <title>Page tables</title>

        <para>HelenOS kernel has two different approaches to the paging

        implementation: <emphasis>4 level page tables</emphasis> and

        <emphasis>global hash tables</emphasis>, which are accessible via

        generic paging abstraction layer. Such different functionality was

        caused by the major architectural differences between supported

        platforms. This abstraction is implemented with help of the global

        structure of pointers to basic mapping functions

        <emphasis>page_mapping_operations</emphasis>. To achieve different

        functionality of page tables, corresponding layer must implement

        functions, declared in

        <emphasis>page_mapping_operations</emphasis></para>

        <formalpara>

          <title>4-level page tables</title>

          <para>4-level page tables are the generalization of the hardware

          capabilities of several architectures.<itemizedlist>

              <listitem>

                 ia32 uses 2-level page tables, with full hardware support.

              </listitem>

              <listitem>

                 amd64 uses 4-level page tables, also coming with full hardware support.

              </listitem>

              <listitem>

                 mips and ppc32 have 2-level tables, software simulated support.

              </listitem>

            </itemizedlist></para>

        </formalpara>

        <formalpara>

          <title>Global hash tables</title>

          <para>- global page hash table: existuje jen jedna v celem systemu

          (vyuziva ji ia64), pozn. ia64 ma zatim vypnuty VHPT. Pouziva se

          genericke hash table s oddelenymi collision chains. ASID support is

          required to use global hash tables.</para>

        </formalpara>

        <para>Thanks to the abstract paging interface, there is possibility

        left have more paging implementations, for example B-Tree page

        tables.</para>

      </section>

      <section id="tlb">

        <title>Translation Lookaside buffer</title>

        <para>Due to the extensive overhead during the page mapping lookup in

        the page tables, all architectures has fast assotiative cache memory

        built-in CPU. This memory called TLB stores recently used page table

        entries.</para>

        <section id="tlb_shootdown">

          <title>TLB consistency. TLB shootdown algorithm.</title>

          <para>Operating system is responsible for keeping TLB consistent by

          invalidating the contents of TLB, whenever there is some change in

          page tables. Those changes may occur when page or group of pages

          were unmapped, mapping is changed or system switching active address

          space to schedule a new system task (which is a batch unmap of all

          address space mappings). Moreover, this invalidation operation must

          be done an all system CPUs because each CPU has its own independent

          TLB cache. Thus maintaining TLB consistency on SMP configuration as

          not as trivial task as it looks at the first glance. Naive solution

          would assume remote TLB invalidatation, which is not possible on the

          most of the architectures, because of the simple fact - flushing TLB

          is allowed only on the local CPU and there is no possibility to

          access other CPUs' TLB caches.</para>

          <para>Technique of remote invalidation of TLB entries is called "TLB

          shootdown". HelenOS uses a variation of the algorithm described by

          D. Black et al., "Translation Lookaside Buffer Consistency: A

          Software Approach," Proc. Third Int'l Conf. Architectural Support

          for Programming Languages and Operating Systems, 1989, pp.

          113-122.</para>

          <para>As the situation demands, you will want partitial invalidation

          of TLB caches. In case of simple memory mapping change it is

          necessary to invalidate only one or more adjacent pages. In case if

          the architecture is aware of ASIDs, during the address space

          switching, kernel invalidates only entries from this particular

          address space. Final option of the TLB invalidation is the complete

          TLB cache invalidation, which is the operation that flushes all

          entries in TLB.</para>

          <para>TLB shootdown is performed in two phases. First, the initiator

          process sends an IPI message indicating the TLB shootdown request to

          the rest of the CPUs. Then, it waits until all CPUs confirm TLB

          invalidating action execution.</para>

        </section>

      </section>

    </section>

    <section>

      <title>---</title>

      <para>At the moment HelenOS does not support swapping.</para>

      <para>- pouzivame vypadky stranky k alokaci ramcu on-demand v ramci

      as_area - na architekturach, ktere to podporuji, podporujeme non-exec

      stranky</para>

    </section>

  </section>

</chapter>