WebSVN – HelenOS-doc – Diff – /design/trunk/src/ch_memory_management.xml


<?xml version="1.0" encoding="UTF-8"?>
<chapter id="mm">
  <?dbhtml filename="mm.html"?>
 
  <title>Memory management</title>
 
  <para>In previous chapters, this book described the scheduling subsystem as
  the creator of the impression that threads execute in parallel. The memory
  management subsystem, on the other hand, creates the impression that there
  is enough physical memory for the kernel and that userspace tasks have the
  entire address space only for themselves.</para>
 
  <section>
    <title>Physical memory management</title>
 
    <section id="zones_and_frames">
      <title>Zones and frames</title>
 
      <para>HelenOS represents continuous areas of physical memory in
      structures called frame zones (abbreviated as zones). Each zone contains
      information about the number of allocated and unallocated physical
      memory frames as well as the physical base address of the zone and
      number of frames contained in it. A zone also contains an array of frame
      structures describing each frame of the zone and, in the last, but not
      the least important, front, each zone is equipped with a buddy system
      that faciliates effective allocation of power-of-two sized block of
      frames.</para>
 
      <para>This organization of physical memory provides good preconditions
      for hot-plugging of more zones. There is also one currently unused zone
      attribute: <code>flags</code>. The attribute could be used to give a
      special meaning to some zones in the future.</para>
 
      <para>The zones are linked in a doubly-linked list. This might seem a
      bit ineffective because the zone list is walked everytime a frame is
      allocated or deallocated. However, this does not represent a significant
      performance problem as it is expected that the number of zones will be
      rather low. Moreover, most architectures merge all zones into
      one.</para>
 
      <para>For each physical memory frame found in a zone, there is a frame
      structure that contains number of references and data used by buddy
      system.</para>
    </section>
 
    <section id="frame_allocator">
      <title>Frame allocator</title>
 
      <para>The frame allocator satisfies kernel requests to allocate
      power-of-two sized blocks of physical memory. Because of zonal
      organization of physical memory, the frame allocator is always working
      within a context of some frame zone. In order to carry out the
      allocation requests, the frame allocator is tightly integrated with the
      buddy system belonging to the zone. The frame allocator is also
      responsible for updating information about the number of free and busy
      frames in the zone. <figure>
          <mediaobject id="frame_alloc">
            <imageobject role="html">
              <imagedata fileref="images/frame_alloc.png" format="PNG" />
            </imageobject>
 
            <imageobject role="fop">
              <imagedata fileref="images.vector/frame_alloc.svg" format="SVG" />
            </imageobject>
          </mediaobject>
 
          <title>Frame allocator scheme.</title>
        </figure></para>
 
      <formalpara>
        <title>Allocation / deallocation</title>
 
        <para>Upon allocation request via function <code>frame_alloc</code>,
        the frame allocator first tries to find a zone that can satisfy the
        request (i.e. has the required amount of free frames). Once a suitable
        zone is found, the frame allocator uses the buddy allocator on the
        zone's buddy system to perform the allocation. During deallocation,
        which is triggered by a call to <code>frame_free</code>, the frame
        allocator looks up the respective zone that contains the frame being
        deallocated. Afterwards, it calls the buddy allocator again, this time
        to take care of deallocation within the zone's buddy system.</para>
      </formalpara>
    </section>
 
    <section id="buddy_allocator">
      <title>Buddy allocator</title>
 
      <para>In the buddy system, the memory is broken down into power-of-two
      sized naturally aligned blocks. These blocks are organized in an array
      of lists, in which the list with index i contains all unallocated blocks
      of size <mathphrase>2<superscript>i</superscript></mathphrase>. The
      index i is called the order of block. Should there be two adjacent
      equally sized blocks in the list i<mathphrase />(i.e. buddies), the
      buddy allocator would coalesce them and put the resulting block in list
      <mathphrase>i + 1</mathphrase>, provided that the resulting block would
      be naturally aligned. Similarily, when the allocator is asked to
      allocate a block of size
      <mathphrase>2<superscript>i</superscript></mathphrase>, it first tries
      to satisfy the request from the list with index i. If the request cannot
      be satisfied (i.e. the list i is empty), the buddy allocator will try to
      allocate and split a larger block from the list with index i + 1. Both
      of these algorithms are recursive. The recursion ends either when there
      are no blocks to coalesce in the former case or when there are no blocks
      that can be split in the latter case.</para>
 
      <para>This approach greatly reduces external fragmentation of memory and
      helps in allocating bigger continuous blocks of memory aligned to their
      size. On the other hand, the buddy allocator suffers increased internal
      fragmentation of memory and is not suitable for general kernel
      allocations. This purpose is better addressed by the <link
      linkend="slab">slab allocator</link>.<figure>
          <mediaobject id="buddy_alloc">
            <imageobject role="html">
              <imagedata fileref="images/buddy_alloc.png" format="PNG" />
            </imageobject>
 
            <imageobject role="fop">
              <imagedata fileref="images.vector/buddy_alloc.svg" format="SVG" />
            </imageobject>
          </mediaobject>
 
          <title>Buddy system scheme.</title>
        </figure></para>
 
      <section>
        <title>Implementation</title>
 
        <para>The buddy allocator is, in fact, an abstract framework wich can
        be easily specialized to serve one particular task. It knows nothing
        about the nature of memory it helps to allocate. In order to beat the
        lack of this knowledge, the buddy allocator exports an interface that
        each of its clients is required to implement. When supplied with an
        implementation of this interface, the buddy allocator can use
        specialized external functions to find a buddy for a block, split and
        coalesce blocks, manipulate block order and mark blocks busy or
        available.</para>
 
        <formalpara>
          <title>Data organization</title>
 
          <para>Each entity allocable by the buddy allocator is required to
          contain space for storing block order number and a link variable
          used to interconnect blocks within the same order.</para>
 
          <para>Whatever entities are allocated by the buddy allocator, the
          first entity within a block is used to represent the entire block.
          The first entity keeps the order of the whole block. Other entities
          within the block are assigned the magic value
          <constant>BUDDY_INNER_BLOCK</constant>. This is especially important
          for effective identification of buddies in a one-dimensional array
          because the entity that represents a potential buddy cannot be
          associated with <constant>BUDDY_INNER_BLOCK</constant> (i.e. if it
          is associated with <constant>BUDDY_INNER_BLOCK</constant> then it is
          not a buddy).</para>
        </formalpara>
      </section>
    </section>
 
    <section id="slab">
      <title>Slab allocator</title>
 
      <para>The majority of memory allocation requests in the kernel is for
      small, frequently used data structures. The basic idea behind the slab
      allocator is that commonly used objects are preallocated in continuous
      areas of physical memory called slabs<footnote>
          <para>Slabs are in fact blocks of physical memory frames allocated
          from the frame allocator.</para>
 
        </footnote>. Whenever an object is to be allocated, the slab allocator
      returns the first available item from a suitable slab corresponding to
      the object type<footnote>
          <para>The mechanism is rather more complicated, see the next
          paragraph.</para>
        </footnote>. Due to the fact that the sizes of the requested and
      allocated object match, the slab allocator significantly freduces
      internal fragmentation.</para>
 
      <para>Slabs of one object type are organized in a structure called slab
      cache. There are ususally more slabs in the slab cache, depending on
      previous allocations. If the the slab cache runs out of available slabs,
      new slabs are allocated. In order to exploit parallelism and to avoid
      locking of shared spinlocks, slab caches can have variants of
      CPU-private slabs called magazines. Each object begins its life in a
      slab. When it is allocated from there, the slab allocator calls a
      constructor that is registered in the respective slab cache. The
      constructor initializes and brings the object into a known state. The
      object is then used by the user. When the user later frees the object,
      the slab allocator puts it into a CPU-private magazine, from where it
      can be precedently allocated again. Note that allocations satisfied from
      a magazine are already initialized by the constructor.</para>
 
      <para>Should HelenOS run short of memory, it would start deallocating
      objects from magazines, calling slab cache destructor on them and
      putting them back into slabs. When a slab contanins no allocated object,
      it is immediately freed.</para>
 
      <para><figure>
          <mediaobject id="slab_alloc">
            <imageobject role="html">
              <imagedata fileref="images/slab_alloc.png" format="PNG" />
            </imageobject>
 
 
 
 
          </mediaobject>
 
          <title>Slab allocator scheme.</title>
        </figure></para>
 
      <section>
 
 
 
 
 
 
 
 
 
 
        <title>Implementation</title>
 
        <para>The slab allocator is closely modelled after <ulink
        url="http://www.usenix.org/events/usenix01/full_papers/bonwick/bonwick_html/">
        OpenSolaris slab allocator by Jeff Bonwick and Jonathan Adams </ulink>
        with the following exceptions:<itemizedlist>
 
 
            <listitem>
               empty magazines are deallocated when not needed
            </listitem>
          </itemizedlist> Following features are not currently supported but
        would be easy to do: <itemizedlist>
            <listitem>
               cache coloring
            </listitem>
 
            <listitem>
               dynamic magazine grow (different magazine sizes are already supported, but we would need to adjust allocation strategy)
            </listitem>
          </itemizedlist></para>
 
        <section>
          <title>Magazine layer</title>
 
          <para>Due to the extensive bottleneck on SMP architures, caused by
          global slab locking mechanism, making processing of all slab
          allocation requests serialized, a new layer was introduced to the
          classic slab allocator design. Slab allocator was extended to
          support per-CPU caches 'magazines' to achieve good SMP scaling.
          <termdef>Slab SMP perfromance bottleneck was resolved by introducing
          a per-CPU caching scheme called as <glossterm>magazine
          layer</glossterm></termdef>.</para>
 
          <para>Magazine is a N-element cache of objects, so each magazine can
          satisfy N allocations. Magazine behaves like a automatic weapon
          magazine (LIFO, stack), so the allocation/deallocation become simple
          push/pop pointer operation. Trick is that CPU does not access global
          slab allocator data during the allocation from its magazine, thus
          making possible parallel allocations between CPUs.</para>
 
          <para>Implementation also requires adding another feature as the
          CPU-bound magazine is actually a pair of magazines to avoid
          thrashing when during allocation/deallocatiion of 1 item at the
          magazine size boundary. LIFO order is enforced, which should avoid
          fragmentation as much as possible.</para>
 
          <para>Another important entity of magazine layer is the common full
          magazine list (also called a depot), that stores full magazines that
          may be used by any of the CPU magazine caches to reload active CPU
          magazine. This list of magazines can be pre-filled with full
          magazines during initialization, but in current implementation it is
          filled during object deallocation, when CPU magazine becomes
          full.</para>
 
          <para>Slab allocator control structures are allocated from special
          slabs, that are marked by special flag, indicating that it should
          not be used for slab magazine layer. This is done to avoid possible
          infinite recursions and deadlock during conventional slab allocaiton
          requests.</para>
        </section>
 
        <section>
          <title>Allocation/deallocation</title>
 
          <para>Every cache contains list of full slabs and list of partialy
          full slabs. Empty slabs are immediately freed (thrashing will be
          avoided because of magazines).</para>
 
          <para>The SLAB allocator allocates lots of space and does not free
          it. When frame allocator fails to allocate the frame, it calls
          slab_reclaim(). It tries 'light reclaim' first, then brutal reclaim.
          The light reclaim releases slabs from cpu-shared magazine-list,
          until at least 1 slab is deallocated in each cache (this algorithm
          should probably change). The brutal reclaim removes all cached
          objects, even from CPU-bound magazines.</para>
 
          <formalpara>
            <title>Allocation</title>
 
            <para><emphasis>Step 1.</emphasis> When it comes to the allocation
            request, slab allocator first of all checks availability of memory
            in local CPU-bound magazine. If it is there, we would just "pop"
            the CPU magazine and return the pointer to object.</para>
 
            <para><emphasis>Step 2.</emphasis> If the CPU-bound magazine is
            empty, allocator will attempt to reload magazin, swapping it with
            second CPU magazine and returns to the first step.</para>
 
            <para><emphasis>Step 3.</emphasis> Now we are in the situation
            when both CPU-bound magazines are empty, which makes allocator to
            access shared full-magazines depot to reload CPU-bound magazines.
            If reload is succesful (meaning there are full magazines in depot)
            algoritm continues at Step 1.</para>
 
            <para><emphasis>Step 4.</emphasis> Final step of the allocation.
            In this step object is allocated from the conventional slab layer
            and pointer is returned.</para>
          </formalpara>
 
          <formalpara>
            <title>Deallocation</title>
 
            <para><emphasis>Step 1.</emphasis> During deallocation request,
            slab allocator will check if the local CPU-bound magazine is not
            full. In this case we will just push the pointer to this
            magazine.</para>
 
            <para><emphasis>Step 2.</emphasis> If the CPU-bound magazine is
            full, allocator will attempt to reload magazin, swapping it with
            second CPU magazine and returns to the first step.</para>
 
            <para><emphasis>Step 3.</emphasis> Now we are in the situation
            when both CPU-bound magazines are full, which makes allocator to
            access shared full-magazines depot to put one of the magazines to
            the depot and creating new empty magazine. Algoritm continues at
            Step 1.</para>
          </formalpara>
        </section>
      </section>
    </section>
 
    <!-- End of Physmem -->
  </section>
 
  <section>
    <title>Virtual memory management</title>
 
    <section>
      <title>Introduction</title>
 
      <para>Virtual memory is a special memory management technique, used by
      kernel to achieve a bunch of mission critical goals. <itemizedlist>
          <listitem>
             Isolate each task from other tasks that are running on the system at the same time.
          </listitem>
 
          <listitem>
             Allow to allocate more memory, than is actual physical memory size of the machine.
          </listitem>
 
          <listitem>
             Allowing, in general, to load and execute two programs that are linked on the same address without complicated relocations.
          </listitem>
        </itemizedlist></para>
 
      <para><!--
 
                TLB shootdown ASID/ASID:PAGE/ALL.
                TLB shootdown requests can come in asynchroniously
                so there is a cache of TLB shootdown requests. Upon cache overflow TLB shootdown ALL is executed
 
 
                <para>
                        Address spaces. Address space area (B+ tree). Only for uspace. Set of syscalls (shrink/extend etc).
                        Special address space area type - device - prohibits shrink/extend syscalls to call on it.
                        Address space has link to mapping tables (hierarchical - per Address space, hash - global tables).
                </para>
 
--></para>
    </section>
 
    <section>
      <title>Paging</title>
 
      <para>Virtual memory is usually using paged memory model, where virtual
      memory address space is divided into the <emphasis>pages</emphasis>
      (usually having size 4096 bytes) and physical memory is divided into the
      frames (same sized as a page, of course). Each page may be mapped to
      some frame and then, upon memory access to the virtual address, CPU
      performs <emphasis>address translation</emphasis> during the instruction
      execution. Non-existing mapping generates page fault exception, calling
      kernel exception handler, thus allowing kernel to manipulate rules of
      memory access. Information for pages mapping is stored by kernel in the
      <link linkend="page_tables">page tables</link></para>
 
      <para>The majority of the architectures use multi-level page tables,
      which means need to access physical memory several times before getting
      physical address. This fact would make serios performance overhead in
      virtual memory management. To avoid this <link linkend="tlb">Traslation
      Lookaside Buffer (TLB)</link> is used.</para>
    </section>
 
    <section>
      <title>Address spaces</title>
 
      <section>
        <title>Address space areas</title>
 
        <para>Each address space consists of mutually disjunctive continuous
        address space areas. Address space area is precisely defined by its
        base address and the number of frames/pages is contains.</para>
 
        <para>Address space area , that define behaviour and permissions on
        the particular area. <itemizedlist>
            <listitem>
               
 
              <emphasis>AS_AREA_READ</emphasis>
 
               flag indicates reading permission.
            </listitem>
 
            <listitem>
               
 
              <emphasis>AS_AREA_WRITE</emphasis>
 
               flag indicates writing permission.
            </listitem>
 
            <listitem>
               
 
              <emphasis>AS_AREA_EXEC</emphasis>
 
               flag indicates code execution permission. Some architectures do not support execution persmission restriction. In this case this flag has no effect.
            </listitem>
 
            <listitem>
               
 
              <emphasis>AS_AREA_DEVICE</emphasis>
 
               marks area as mapped to the device memory.
            </listitem>
          </itemizedlist></para>
 
        <para>Kernel provides possibility tasks create/expand/shrink/share its
        address space via the set of syscalls.</para>
      </section>
 
      <section>
        <title>Address Space ID (ASID)</title>
 
        <para>When switching to the different task, kernel also require to
        switch mappings to the different address space. In case TLB cannot
        distinguish address space mappings, all mapping information in TLB
        from the old address space must be flushed, which can create certain
        uncessary overhead during the task switching. To avoid this, some
        architectures have capability to segregate different address spaces on
        hardware level introducing the address space identifier as a part of
        TLB record, telling the virtual address space translation unit to
        which address space this record is applicable.</para>
 
        <para>HelenOS kernel can take advantage of this hardware supported
        identifier by having an ASID abstraction which is somehow related to
        the corresponding architecture identifier. I.e. on ia64 kernel ASID is
        derived from RID (region identifier) and on the mips32 kernel ASID is
        actually the hardware identifier. As expected, this ASID information
        record is the part of <emphasis>as_t</emphasis> structure.</para>
 
        <para>Due to the hardware limitations, hardware ASID has limited
        length from 8 bits on ia64 to 24 bits on mips32, which makes it
        impossible to use it as unique address space identifier for all tasks
        running in the system. In such situations special ASID stealing
        algoritm is used, which takes ASID from inactive task and assigns it
        to the active task.</para>
 
        <para><classname>ASID stealing algoritm here.</classname></para>
      </section>
    </section>
 
    <section>
      <title>Virtual address translation</title>
 
      <section id="page_tables">
        <title>Page tables</title>
 
        <para>HelenOS kernel has two different approaches to the paging
        implementation: <emphasis>4 level page tables</emphasis> and
        <emphasis>global hash tables</emphasis>, which are accessible via
        generic paging abstraction layer. Such different functionality was
        caused by the major architectural differences between supported
        platforms. This abstraction is implemented with help of the global
        structure of pointers to basic mapping functions
        <emphasis>page_mapping_operations</emphasis>. To achieve different
        functionality of page tables, corresponding layer must implement
        functions, declared in
        <emphasis>page_mapping_operations</emphasis></para>
 
        <formalpara>
          <title>4-level page tables</title>
 
          <para>4-level page tables are the generalization of the hardware
          capabilities of several architectures.<itemizedlist>
              <listitem>
                 ia32 uses 2-level page tables, with full hardware support.
              </listitem>
 
              <listitem>
                 amd64 uses 4-level page tables, also coming with full hardware support.
              </listitem>
 
              <listitem>
                 mips and ppc32 have 2-level tables, software simulated support.
              </listitem>
            </itemizedlist></para>
        </formalpara>
 
        <formalpara>
          <title>Global hash tables</title>
 
          <para>- global page hash table: existuje jen jedna v celem systemu
          (vyuziva ji ia64), pozn. ia64 ma zatim vypnuty VHPT. Pouziva se
          genericke hash table s oddelenymi collision chains. ASID support is
          required to use global hash tables.</para>
        </formalpara>
 
        <para>Thanks to the abstract paging interface, there is possibility
        left have more paging implementations, for example B-Tree page
        tables.</para>
      </section>
 
      <section id="tlb">
        <title>Translation Lookaside buffer</title>
 
        <para>Due to the extensive overhead during the page mapping lookup in
        the page tables, all architectures has fast assotiative cache memory
        built-in CPU. This memory called TLB stores recently used page table
        entries.</para>
 
        <section id="tlb_shootdown">
          <title>TLB consistency. TLB shootdown algorithm.</title>
 
          <para>Operating system is responsible for keeping TLB consistent by
          invalidating the contents of TLB, whenever there is some change in
          page tables. Those changes may occur when page or group of pages
          were unmapped, mapping is changed or system switching active address
          space to schedule a new system task (which is a batch unmap of all
          address space mappings). Moreover, this invalidation operation must
          be done an all system CPUs because each CPU has its own independent
          TLB cache. Thus maintaining TLB consistency on SMP configuration as
          not as trivial task as it looks at the first glance. Naive solution
          would assume remote TLB invalidatation, which is not possible on the
          most of the architectures, because of the simple fact - flushing TLB
          is allowed only on the local CPU and there is no possibility to
          access other CPUs' TLB caches.</para>
 
          <para>Technique of remote invalidation of TLB entries is called "TLB
          shootdown". HelenOS uses a variation of the algorithm described by
          D. Black et al., "Translation Lookaside Buffer Consistency: A
          Software Approach," Proc. Third Int'l Conf. Architectural Support
          for Programming Languages and Operating Systems, 1989, pp.
          113-122.</para>
 
          <para>As the situation demands, you will want partitial invalidation
          of TLB caches. In case of simple memory mapping change it is
          necessary to invalidate only one or more adjacent pages. In case if
          the architecture is aware of ASIDs, during the address space
          switching, kernel invalidates only entries from this particular
          address space. Final option of the TLB invalidation is the complete
          TLB cache invalidation, which is the operation that flushes all
          entries in TLB.</para>
 
          <para>TLB shootdown is performed in two phases. First, the initiator
          process sends an IPI message indicating the TLB shootdown request to
          the rest of the CPUs. Then, it waits until all CPUs confirm TLB
          invalidating action execution.</para>
        </section>
      </section>
    </section>
 
    <section>
      <title>---</title>
 
      <para>At the moment HelenOS does not support swapping.</para>
 
      <para>- pouzivame vypadky stranky k alokaci ramcu on-demand v ramci
      as_area - na architekturach, ktere to podporuji, podporujeme non-exec
      stranky</para>
    </section>
  </section>
</chapter>
Subversion Repositories HelenOS-doc

(root)/design/trunk/src/ch_memory_management.xml @ 185 – Rev 64 → 65
