WebSVN – HelenOS-doc – Diff – /design/trunk/src/ch_memory_management.xml


<?xml version="1.0" encoding="UTF-8"?>
<chapter id="mm">
  <?dbhtml filename="mm.html"?>
 
  <title>Memory management</title>
 
  <para>In previous chapters, this book described the scheduling subsystem as
  the creator of the impression that threads execute in parallel. The memory
  management subsystem, on the other hand, creates the impression that there
  is enough physical memory for the kernel and that userspace tasks have the
  entire address space only for themselves.</para>
 
  <section>
    <title>Physical memory management</title>
 
    <section id="zones_and_frames">
      <title>Zones and frames</title>
 
      <para>HelenOS represents continuous areas of physical memory in
      structures called frame zones (abbreviated as zones). Each zone contains
      information about the number of allocated and unallocated physical
      memory frames as well as the physical base address of the zone and
      number of frames contained in it. A zone also contains an array of frame
      structures describing each frame of the zone and, in the last, but not
      the least important, front, each zone is equipped with a buddy system
      that faciliates effective allocation of power-of-two sized block of
      frames.</para>
 
      <para>This organization of physical memory provides good preconditions
      for hot-plugging of more zones. There is also one currently unused zone
      attribute: <code>flags</code>. The attribute could be used to give a
      special meaning to some zones in the future.</para>
 
      <para>The zones are linked in a doubly-linked list. This might seem a
      bit ineffective because the zone list is walked everytime a frame is
      allocated or deallocated. However, this does not represent a significant
      performance problem as it is expected that the number of zones will be
      rather low. Moreover, most architectures merge all zones into
      one.</para>
 
      <para>Every physical memory frame in a zone, is described by a structure
      that contains number of references and other data used by buddy
      system.</para>
    </section>
 
    <section id="frame_allocator">
      <indexterm>
        <primary>frame allocator</primary>
      </indexterm>
 
      <title>Frame allocator</title>
 
      <para>The frame allocator satisfies kernel requests to allocate
      power-of-two sized blocks of physical memory. Because of zonal
      organization of physical memory, the frame allocator is always working
      within a context of a particular frame zone. In order to carry out the
      allocation requests, the frame allocator is tightly integrated with the
      buddy system belonging to the zone. The frame allocator is also
      responsible for updating information about the number of free and busy
      frames in the zone. <figure>
          <mediaobject id="frame_alloc">
            <imageobject role="eps">
              <imagedata fileref="images.vector/frame_alloc.eps" format="EPS" />
            </imageobject>
 
            <imageobject role="html">
              <imagedata fileref="images/frame_alloc.png" format="PNG" />
            </imageobject>
 
            <imageobject role="fop">
              <imagedata fileref="images.vector/frame_alloc.svg" format="SVG" />
            </imageobject>
          </mediaobject>
 
          <title>Frame allocator scheme.</title>
        </figure></para>
 
      <formalpara>
        <title>Allocation / deallocation</title>
 
        <para>Upon allocation request via function <code>frame_alloc</code>,
        the frame allocator first tries to find a zone that can satisfy the
        request (i.e. has the required amount of free frames). Once a suitable
        zone is found, the frame allocator uses the buddy allocator on the
        zone's buddy system to perform the allocation. During deallocation,
        which is triggered by a call to <code>frame_free</code>, the frame
        allocator looks up the respective zone that contains the frame being
        deallocated. Afterwards, it calls the buddy allocator again, this time
        to take care of deallocation within the zone's buddy system.</para>
      </formalpara>
    </section>
 
    <section id="buddy_allocator">
      <indexterm>
        <primary>buddy system</primary>
      </indexterm>
 
      <title>Buddy allocator</title>
 
      <para>In the buddy system, the memory is broken down into power-of-two
      sized naturally aligned blocks. These blocks are organized in an array
      of lists, in which the list with index i contains all unallocated blocks
      of size <mathphrase>2<superscript>i</superscript></mathphrase>. The
      index i is called the order of block. Should there be two adjacent
      equally sized blocks in the list i<mathphrase />(i.e. buddies), the
      buddy allocator would coalesce them and put the resulting block in list
      <mathphrase>i + 1</mathphrase>, provided that the resulting block would
      be naturally aligned. Similarily, when the allocator is asked to
      allocate a block of size
      <mathphrase>2<superscript>i</superscript></mathphrase>, it first tries
      to satisfy the request from the list with index i. If the request cannot
      be satisfied (i.e. the list i is empty), the buddy allocator will try to
      allocate and split a larger block from the list with index i + 1. Both
      of these algorithms are recursive. The recursion ends either when there
      are no blocks to coalesce in the former case or when there are no blocks
      that can be split in the latter case.</para>
 
      <para>This approach greatly reduces external fragmentation of memory and
      helps in allocating bigger continuous blocks of memory aligned to their
      size. On the other hand, the buddy allocator suffers increased internal
      fragmentation of memory and is not suitable for general kernel
      allocations. This purpose is better addressed by the <link
      linkend="slab">slab allocator</link>.<figure>
          <mediaobject id="buddy_alloc">
            <imageobject role="eps">
              <imagedata fileref="images.vector/buddy_alloc.eps" format="EPS" />
            </imageobject>
 
            <imageobject role="html">
              <imagedata fileref="images/buddy_alloc.png" format="PNG" />
            </imageobject>
 
            <imageobject role="fop">
              <imagedata fileref="images.vector/buddy_alloc.svg" format="SVG" />
            </imageobject>
          </mediaobject>
 
          <title>Buddy system scheme.</title>
        </figure></para>
 
      <section>
        <title>Implementation</title>
 
        <para>The buddy allocator is, in fact, an abstract framework wich can
        be easily specialized to serve one particular task. It knows nothing
        about the nature of memory it helps to allocate. In order to beat the
        lack of this knowledge, the buddy allocator exports an interface that
        each of its clients is required to implement. When supplied with an
        implementation of this interface, the buddy allocator can use
        specialized external functions to find a buddy for a block, split and
        coalesce blocks, manipulate block order and mark blocks busy or
        available.</para>
 
        <formalpara>
          <title>Data organization</title>
 
          <para>Each entity allocable by the buddy allocator is required to
          contain space for storing block order number and a link variable
          used to interconnect blocks within the same order.</para>
 
          <para>Whatever entities are allocated by the buddy allocator, the
          first entity within a block is used to represent the entire block.
          The first entity keeps the order of the whole block. Other entities
          within the block are assigned the magic value
          <constant>BUDDY_INNER_BLOCK</constant>. This is especially important
          for effective identification of buddies in a one-dimensional array
          because the entity that represents a potential buddy cannot be
          associated with <constant>BUDDY_INNER_BLOCK</constant> (i.e. if it
          is associated with <constant>BUDDY_INNER_BLOCK</constant> then it is
          not a buddy).</para>
        </formalpara>
      </section>
    </section>
 
    <section id="slab">
      <indexterm>
        <primary>slab allocator</primary>
      </indexterm>
 
      <title>Slab allocator</title>
 
      <para>The majority of memory allocation requests in the kernel is for
      small, frequently used data structures. The basic idea behind the slab
      allocator is that commonly used objects are preallocated in continuous
      areas of physical memory called slabs<footnote>
          <para>Slabs are in fact blocks of physical memory frames allocated
          from the frame allocator.</para>
        </footnote>. Whenever an object is to be allocated, the slab allocator
      returns the first available item from a suitable slab corresponding to
      the object type<footnote>
          <para>The mechanism is rather more complicated, see the next
          paragraph.</para>
        </footnote>. Due to the fact that the sizes of the requested and
      allocated object match, the slab allocator significantly reduces
      internal fragmentation.</para>
 
      <indexterm>
        <primary>slab allocator</primary>
 
        <secondary>- slab cache</secondary>
      </indexterm>
 
      <para>Slabs of one object type are organized in a structure called slab
      cache. There are ususally more slabs in the slab cache, depending on
      previous allocations. If the the slab cache runs out of available slabs,
      new slabs are allocated. In order to exploit parallelism and to avoid
      locking of shared spinlocks, slab caches can have variants of
      processor-private slabs called magazines. On each processor, there is a
      two-magazine cache. Full magazines that are not part of any
      per-processor magazine cache are stored in a global list of full
      magazines.</para>
 
      <indexterm>
        <primary>slab allocator</primary>
 
        <secondary>- magazine</secondary>
      </indexterm>
 
      <para>Each object begins its life in a slab. When it is allocated from
      there, the slab allocator calls a constructor that is registered in the
      respective slab cache. The constructor initializes and brings the object
      into a known state. The object is then used by the user. When the user
      later frees the object, the slab allocator puts it into a processor
      private <indexterm>
          <primary>slab allocator</primary>
 
          <secondary>- magazine</secondary>
        </indexterm>magazine cache, from where it can be precedently allocated
      again. Note that allocations satisfied from a magazine are already
      initialized by the constructor. When both of the processor cached
      magazines get full, the allocator will move one of the magazines to the
      list of full magazines. Similarily, when allocating from an empty
      processor magazine cache, the kernel will reload only one magazine from
      the list of full magazines. In other words, the slab allocator tries to
      keep the processor magazine cache only half-full in order to prevent
      thrashing when allocations and deallocations interleave on magazine
      boundaries. The advantage of this setup is that during most of the
      allocations, no global spinlock needs to be held.</para>
 
      <para>Should HelenOS run short of memory, it would start deallocating
      objects from magazines, calling slab cache destructor on them and
      putting them back into slabs. When a slab contanins no allocated object,
      it is immediately freed.</para>
 
      <para>
        <figure>
          <mediaobject id="slab_alloc">
            <imageobject role="eps">
              <imagedata fileref="images.vector/slab_alloc.eps" format="EPS" />
            </imageobject>
 
            <imageobject role="html">
              <imagedata fileref="images/slab_alloc.png" format="PNG" />
            </imageobject>
 
            <imageobject role="fop">
              <imagedata fileref="images.vector/slab_alloc.svg" format="SVG" />
            </imageobject>
          </mediaobject>
 
          <title>Slab allocator scheme.</title>
        </figure>
      </para>
 
      <section>
        <title>Implementation</title>
 
        <para>The slab allocator is closely modelled after OpenSolaris slab
        allocator by Jeff Bonwick and Jonathan Adams <xref
        linkend="Bonwick01" /> with the following exceptions:<itemizedlist>
            <listitem>empty slabs are immediately deallocated and</listitem>
 
            <listitem>
              <para>empty magazines are deallocated when not needed.</para>
            </listitem>
          </itemizedlist>The following features are not currently supported
        but would be easy to do: <itemizedlist>
            <listitem>cache coloring and</listitem>
 
            <listitem>dynamic magazine grow (different magazine sizes are
            already supported, but the allocation strategy would need to be
            adjusted).</listitem>
          </itemizedlist></para>
 
        <section>
          <title>Allocation/deallocation</title>
 
          <para>The following two paragraphs summarize and complete the
          description of the slab allocator operation (i.e.
          <code>slab_alloc</code> and <code>slab_free</code>
          operations).</para>
 
          <formalpara>
            <title>Allocation</title>
 
            <para><emphasis>Step 1.</emphasis> When an allocation request
            comes, the slab allocator checks availability of memory in the
            current magazine of the local processor magazine cache. If the
            available memory is there, the allocator just pops the object from
            magazine and returns it.</para>
 
            <para><emphasis>Step 2.</emphasis> If the current magazine in the
            processor magazine cache is empty, the allocator will attempt to
            swap it with the last magazine from the cache and return to the
            first step. If also the last magazine is empty, the algorithm will
            fall through to Step 3.</para>
 
            <para><emphasis>Step 3.</emphasis> Now the allocator is in the
            situation when both magazines in the processor magazine cache are
            empty. The allocator reloads one magazine from the shared list of
            full magazines. If the reload is successful (i.e. there are full
            magazines in the list), the algorithm continues with Step
            1.</para>
 
            <para><emphasis>Step 4.</emphasis> In this fail-safe step, an
            object is allocated from the conventional slab layer and a pointer
            to it is returned. If also the last magazine is full,</para>
          </formalpara>
 
          <formalpara>
            <title>Deallocation</title>
 
            <para><emphasis>Step 1.</emphasis> During a deallocation request,
            the slab allocator checks if the current magazine of the local
            processor magazine cache is not full. If it is, the pointer to the
            objects is just pushed into the magazine and the algorithm
            returns.</para>
 
            <para><emphasis>Step 2.</emphasis> If the current magazine is
            full, the allocator will attempt to swap it with the last magazine
            from the cache and return to the first step. If also the last
            magazine is empty, the algorithm will fall through to Step
            3.</para>
 
            <para><emphasis>Step 3.</emphasis> Now the allocator is in the
            situation when both magazines in the processor magazine cache are
            full. The allocator tries to allocate a new empty magazine and
            flush one of the full magazines to the shared list of full
            magazines. If it is successfull, the algoritm continues with Step
            1.</para>
 
            <para><emphasis>Step 4. </emphasis>In case of low memory condition
            when the allocation of empty magazine fails, the object is moved
            directly into slab. In the worst case object deallocation does not
            need to allocate any additional memory.</para>
          </formalpara>
        </section>
      </section>
    </section>
  </section>
 
  <section>
    <title>Virtual memory management</title>
 
    <section>
      <title>Introduction</title>
 
      <para>Virtual memory is a special memory management technique, used by
      kernel to achieve a bunch of mission critical goals. <itemizedlist>
          <listitem>
             Isolate each task from other tasks that are running on the system at the same time.
          </listitem>
 
          <listitem>
             Allow to allocate more memory, than is actual physical memory size of the machine.
          </listitem>
 
          <listitem>
             Allowing, in general, to load and execute two programs that are linked on the same address without complicated relocations.
          </listitem>
        </itemizedlist></para>
 
      <para><!--
                <para>
                        Address spaces. Address space area (B+ tree). Only for uspace. Set of syscalls (shrink/extend etc).
                        Special address space area type - device - prohibits shrink/extend syscalls to call on it.
                        Address space has link to mapping tables (hierarchical - per Address space, hash - global tables).
                </para>
 
--></para>
    </section>
 
    <section>
      <title>Address spaces</title>
 
      <section>
        <indexterm>
          <primary>address space</primary>
 
          <secondary>- area</secondary>
        </indexterm>
 
        <title>Address space areas</title>
 
        <para>Each address space consists of mutually disjunctive continuous
        address space areas. Address space area is precisely defined by its
        base address and the number of frames/pages is contains.</para>
 
        <para>Address space area , that define behaviour and permissions on
        the particular area. <itemizedlist>
            <listitem><emphasis>AS_AREA_READ</emphasis> flag indicates reading
            permission.</listitem>
 
            <listitem><emphasis>AS_AREA_WRITE</emphasis> flag indicates
            writing permission.</listitem>
 
            <listitem><emphasis>AS_AREA_EXEC</emphasis> flag indicates code
            execution permission. Some architectures do not support execution
            persmission restriction. In this case this flag has no
            effect.</listitem>
 
            <listitem><emphasis>AS_AREA_DEVICE</emphasis> marks area as mapped
            to the device memory.</listitem>
          </itemizedlist></para>
 
        <para>Kernel provides possibility tasks create/expand/shrink/share its
        address space via the set of syscalls.</para>
      </section>
 
      <section>
        <indexterm>
          <primary>address space</primary>
 
          <secondary>- ASID</secondary>
        </indexterm>
 
        <title>Address Space ID (ASID)</title>
 
        <para>Every task in the operating system has it's own view of the
        virtual memory. When performing context switch between different
        tasks, the kernel must switch the address space mapping as well. As
        modern processors perform very aggressive caching of virtual mappings,
        flushing the complete TLB on every context switch would be very
        inefficient. To avoid such performance penalty, some architectures
        introduce an address space identifier, which allows storing several
        different mappings inside TLB.</para>
 
        <para>HelenOS kernel can take advantage of this hardware support by
        having an ASID abstraction. I.e. on ia64 kernel ASID is derived from
        RID (region identifier) and on the mips32 kernel ASID is actually the
        hardware identifier. As expected, this ASID information record is the
        part of <emphasis>as_t</emphasis> structure.</para>
 
        <para>Due to the hardware limitations, hardware ASID has limited
        length from 8 bits on ia64 to 24 bits on mips32, which makes it
        impossible to use it as unique address space identifier for all tasks
        running in the system. In such situations special ASID stealing
        algoritm is used, which takes ASID from inactive task and assigns it
        to the active task.</para>
 
        <indexterm>
          <primary>address space</primary>
 
          <secondary>- ASID stealing</secondary>
        </indexterm>
 
        <para>
          <classname>ASID stealing algoritm here.</classname>
        </para>
      </section>
    </section>
 
    <section id="paging">
      <title>Virtual address translation</title>
 
      <section>
        <title>Introduction</title>
 
        <para>Virtual memory is usually using paged memory model, where
        virtual memory address space is divided into the
        <emphasis>pages</emphasis> (usually having size 4096 bytes) and
        physical memory is divided into the frames (same sized as a page, of
        course). Each page may be mapped to some frame and then, upon memory
        access to the virtual address, CPU performs <emphasis>address
        translation</emphasis> during the instruction execution. Non-existing
        mapping generates page fault exception, calling kernel exception
        handler, thus allowing kernel to manipulate rules of memory access.
        Information for pages mapping is stored by kernel in the <link
        linkend="page_tables">page tables</link></para>
 
        <indexterm>
          <primary>page tables</primary>
        </indexterm>
 
        <para>The majority of the architectures use multi-level page tables,
        which means need to access physical memory several times before
        getting physical address. This fact would make serios performance
        overhead in virtual memory management. To avoid this <link
        linkend="tlb">Traslation Lookaside Buffer (TLB)</link> is used.</para>
 
        <para>HelenOS kernel has two different approaches to the paging
        implementation: <emphasis>4 level page tables</emphasis> and
        <emphasis>global hash table</emphasis>, which are accessible via
        generic paging abstraction layer. Such different functionality was
        caused by the major architectural differences between supported
        platforms. This abstraction is implemented with help of the global
        structure of pointers to basic mapping functions
        <emphasis>page_mapping_operations</emphasis>. To achieve different
        functionality of page tables, corresponding layer must implement
        functions, declared in
        <emphasis>page_mapping_operations</emphasis></para>
 
        <para>Thanks to the abstract paging interface, there was a place left
        for more paging implementations (besides already implemented
        hieararchical page tables and hash table), for example <indexterm>
            <primary>B-tree</primary>
          </indexterm> B-Tree based page tables.</para>
      </section>
 
      <section id="page_tables">
        <indexterm>
          <primary>page tables</primary>
 
          <secondary>- hierarchical</secondary>
        </indexterm>
 
        <title>Hierarchical 4-level page tables</title>
 
        <para>Hierarchical 4-level page tables are the generalization of the
        hardware capabilities of most architectures. Each address space has
        its own page tables.<itemizedlist>
            <listitem>ia32 uses 2-level page tables, with full hardware
            support.</listitem>
 
            <listitem>amd64 uses 4-level page tables, also coming with full
            hardware support.</listitem>
 
            <listitem>mips and ppc32 have 2-level tables, software simulated
            support.</listitem>
          </itemizedlist></para>
      </section>
 
      <section>
        <indexterm>
          <primary>page tables</primary>
 
          <secondary>- hashing</secondary>
        </indexterm>
 
        <title>Global hash table</title>
 
        <para>Implementation of the global hash table was encouraged by the
        ia64 architecture support. One of the major differences between global
        hash table and hierarchical tables is that global hash table exists
        only once in the system and the hierarchical tables are maintained per
        address space.</para>
 
        <para>Thus, hash table contains information about all address spaces
        mappings in the system, so, the hash of an entry must contain
        information of both address space pointer or id and the virtual
        address of the page. Generic hash table implementation assumes that
        the addresses of the pointers to the address spaces are likely to be
        on the close addresses, so it uses least significant bits for hash;
        also it assumes that the virtual page addresses have roughly the same
        probability of occurring, so the least significant bits of VPN compose
        the hash index.</para>
 
        <para>Paging hash table uses generic hash table with collision chains
        (see the <link linkend="hashtables">Data Structures</link> chapter of
        this manual for details).</para>
      </section>
    </section>
 
    <section id="tlb">
      <indexterm>
        <primary>TLB</primary>
      </indexterm>
 
      <title>Translation Lookaside buffer</title>
 
      <para>Due to the extensive overhead during the page mapping lookup in
      the page tables, all architectures has fast assotiative cache memory
      built-in CPU. This memory called TLB stores recently used page table
      entries.</para>
 
      <section id="tlb_shootdown">
        <indexterm>
          <primary>TLB</primary>
 
          <secondary>- TLB shootdown</secondary>
        </indexterm>
 
        <title>TLB consistency. TLB shootdown algorithm.</title>
 
        <para>Operating system is responsible for keeping TLB consistent by
        invalidating the contents of TLB, whenever there is some change in
        page tables. Those changes may occur when page or group of pages were
        unmapped, mapping is changed or system switching active address space
        to schedule a new system task. Moreover, this invalidation operation
        must be done an all system CPUs because each CPU has its own
        independent TLB cache. Thus maintaining TLB consistency on SMP
        configuration as not as trivial task as it looks on the first glance.
        Naive solution would assume that is the CPU which wants to invalidate
        TLB will invalidate TLB caches on other CPUs. It is not possible on
        the most of the architectures, because of the simple fact - flushing
        TLB is allowed only on the local CPU and there is no possibility to
        access other CPUs' TLB caches, thus invalidate TLB remotely.</para>
 
        <para>Technique of remote invalidation of TLB entries is called "TLB
        shootdown". HelenOS uses a variation of the algorithm described by D.
        Black et al., "Translation Lookaside Buffer Consistency: A Software
        Approach," Proc. Third Int'l Conf. Architectural Support for
        Programming Languages and Operating Systems, 1989, pp. 113-122. <xref
        linkend="Black89" /></para>
 
        <para>As the situation demands, you will want partitial invalidation
        of TLB caches. In case of simple memory mapping change it is necessary
        to invalidate only one or more adjacent pages. In case if the
        architecture is aware of ASIDs, when kernel needs to dump some ASID to
        use by another task, it invalidates only entries from this particular
        address space. Final option of the TLB invalidation is the complete
        TLB cache invalidation, which is the operation that flushes all
        entries in TLB.</para>
 
        <para>TLB shootdown is performed in two phases.</para>
 
        <formalpara>
          <title>Phase 1.</title>
 
          <para>First, initiator locks a global TLB spinlock, then request is
          being put to the local request cache of every other CPU in the
          system protected by its spinlock. In case the cache is full, all
          requests in the cache are replaced by one request, indicating global
          TLB flush. Then the initiator thread sends an IPI message indicating
          the TLB shootdown request to the rest of the CPUs and waits actively
          until all CPUs confirm TLB invalidating action execution by setting
          up a special flag. After setting this flag this thread is blocked on
          the TLB spinlock, held by the initiator.</para>
        </formalpara>
 
        <formalpara>
          <title>Phase 2.</title>
 
          <para>All CPUs are waiting on the TLB spinlock to execute TLB
          invalidation action and have indicated their intention to the
          initiator. Initiator continues, cleaning up its TLB and releasing
          the global TLB spinlock. After this all other CPUs gain and
          immidiately release TLB spinlock and perform TLB invalidation
          actions.</para>
        </formalpara>
      </section>
    </section>
  </section>
</chapter>
Subversion Repositories HelenOS-doc

(root)/design/trunk/src/ch_memory_management.xml @ 185 – Rev 76 → 77
