WebSVN – HelenOS-doc – Diff – /design/trunk/src/ch_memory_management.xml


<?xml version="1.0" encoding="UTF-8"?>
<chapter id="mm">
  <?dbhtml filename="mm.html"?>
 
  <title>Memory management</title>
 
  <section>
    <title>Virtual memory management</title>
 
    <section>
      <title>Introduction</title>
 
      <para>Virtual memory is a special memory management technique, used by
      kernel to achieve a bunch of mission critical goals. <itemizedlist>
          <listitem>
             Isolate each task from other tasks that are running on the system at the same time.
          </listitem>
 
          <listitem>
             Allow to allocate more memory, than is actual physical memory size of the machine.
          </listitem>
 
          <listitem>
             Allowing, in general, to load and execute two programs that are linked on the same address without complicated relocations.
          </listitem>
        </itemizedlist></para>
 
      <para><!--
 
                TLB shootdown ASID/ASID:PAGE/ALL.
                TLB shootdown requests can come in asynchroniously
                so there is a cache of TLB shootdown requests. Upon cache overflow TLB shootdown ALL is executed
 
 
                <para>
                        Address spaces. Address space area (B+ tree). Only for uspace. Set of syscalls (shrink/extend etc).
                        Special address space area type - device - prohibits shrink/extend syscalls to call on it.
                        Address space has link to mapping tables (hierarchical - per Address space, hash - global tables).
                </para>
 
--></para>
    </section>
 
    <section>
      <title>Paging</title>
 
      <para>Virtual memory is usually using paged memory model, where virtual
      memory address space is divided into the <emphasis>pages</emphasis>
      (usually having size 4096 bytes) and physical memory is divided into the
      frames (same sized as a page, of course). Each page may be mapped to
      some frame and then, upon memory access to the virtual address, CPU
      performs <emphasis>address translation</emphasis> during the instruction
      execution. Non-existing mapping generates page fault exception, calling
      kernel exception handler, thus allowing kernel to manipulate rules of
      memory access. Information for pages mapping is stored by kernel in the
      <link linkend="page_tables">page tables</link></para>
 
      <para>The majority of the architectures use multi-level page tables,
      which means need to access physical memory several times before getting
      physical address. This fact would make serios performance overhead in
      virtual memory management. To avoid this <link linkend="tlb">Traslation
      Lookaside Buffer (TLB)</link> is used.</para>
 
      <para>At the moment HelenOS does not support swapping.</para>
 
      <para>- pouzivame vypadky stranky k alokaci ramcu on-demand v ramci
      as_area - na architekturach, ktere to podporuji, podporujeme non-exec
      stranky</para>
    </section>
 
    <section>
      <title>Address spaces</title>
 
      <section>
        <title>Address spaces and areas</title>
 
        <para>- adresovy prostor se sklada z tzv. address space areas
        usporadanych v B+stromu; tyto areas popisuji vyuzivane casti
        adresoveho prostoru patrici do user address space. Kazda cast je dana
        svoji bazovou adresou, velikosti a flagy (rwx/dd).</para>
 
        <para>- uzivatelske thready maji moznost manipulovat se svym adresovym
        prostorem (vytvaret/resizovat/sdilet) as_areas pomoci syscallu</para>
      </section>
 
      <section>
        <title>Address Space ID (ASID)</title>
 
        <para>- nektery hardware umoznuje rozlisit ruzne adresove prostory od
        sebe (cilem je maximalizovat vyuziti TLB); dela to tak, ze s kazdou
        polozkou TLB/strankovacich tabulek sdruzi identifikator adresoveho
        prostoru (ASID, RID, ppc32 ???). Tyto id mivaji ruznou sirku: 8-bitu
        az 24-bitu (kolik ma ppc32?)</para>
 
        <para>- kernel tomu rozumi a sam pouziva abstrakci ASIDu (na ia64 to
        je napr. cislo odvozene od RIDu, na mips32 to je ASID samotny);
        existence ASIDu je nutnou podminkou pouziti _global_ page hash table
        mechanismu.</para>
 
        <para>- na vsech arch. plati, ze asidu je mnohem mene, nez teoreticky
        pocet soucasne bezicich tasku ~ adresovych prostoru, takze je
        implementovan mechanismus, ktery umoznuje jednomu adresovemu prostoru
        ASID odebrat a pridelit ho jinemu</para>
 
        <para>- vztah task ~ adresovy prostor: teoreticky existuje moznost, ze
        je adresovy prostor sdilen vice tasky, avsak tuto moznost nepouzivame
        a neni ani nijak osetrena. Tim padem plati, ze kazdy task ma vlastni
        adresovy prostor</para>
      </section>
    </section>
 
    <section>
      <title>Virtual address translation</title>
 
      <section id="page_tables">
        <title>Page tables</title>
 
        <para>HelenOS kernel has two different approaches to the paging
        implementation: <emphasis>4 level page tables</emphasis> and
        <emphasis>global hash tables</emphasis>, which are accessible via
        generic paging abstraction layer. This division was caused by the
        major architectural differences between different platforms.</para>
 
        <formalpara>
          <title>4-level page tables</title>
 
          <para>4-level page tables are the generalization of the hardware
          capabilities of the certain platforms. <itemizedlist>
              <listitem>
                 ia32 uses 2-level page tables, with full hardware support.
              </listitem>
 
              <listitem>
                 amd64 uses 4-level page tables, also coming with full hardware support.
              </listitem>
 
              <listitem>
                 mips and ppc32 have 2-level tables, software simulated support.
              </listitem>
            </itemizedlist></para>
        </formalpara>
 
        <formalpara>
          <title>Global hash tables</title>
 
          <para>- global page hash table: existuje jen jedna v celem systemu
          (vyuziva ji ia64), pozn. ia64 ma zatim vypnuty VHPT. Pouziva se
          genericke hash table s oddelenymi collision chains</para>
        </formalpara>
 
        <para>Thanks to the abstract paging interface, there is possibility
        left have more paging implementations, for example B-Tree page
        tables.</para>
      </section>
 
      <section id="tlb">
        <title>Translation Lookaside buffer</title>
 
        <para>- TLB cachuji informace ve strankovacich tabulkach; alternativne
        se lze na strankovaci tabulky (ci ruzne hw rozsireni [e.g. VHPT, ppc32
        hw hash table]) divat jako na velke TLB</para>
 
        <para>- pri modifikaci mapovani nebo odstraneni mapovani ze
        strankovacich tabulek je potreba zajistit konsistenci TLB a techto
        tabulek; nutne delat na vsech CPU; na to mame zjednodusenou verzi TLB
        shootdown mechanismu; je to variace na algoritmus popsany zde: D.
        Black et al., "Translation Lookaside Buffer Consistency: A Software
        Approach," Proc. Third Int'l Conf. Architectural Support for
        Programming Languages and Operating Systems, 1989, pp. 113-122.</para>
 
        <para>- nutno poznamenat, ze existuji odlehcenejsi verze TLB shootdown
        algoritmu</para>
      </section>
    </section>
  </section>
 
  <!-- End of VM -->
 
  <section>
    <!-- Phys mem -->
 
    <title>Physical memory management</title>
 
    <section id="zones_and_frames">
      <title>Zones and frames</title>
 
      <para><!--graphic fileref="images/mm2.png" /--><!--graphic fileref="images/buddy_alloc.svg" format="SVG" /--></para>
 
      <para>On some architectures not whole physical memory is available for
      conventional usage. This limitations require from kernel to maintain a
      table of available and unavailable ranges of physical memory addresses.
      Main idea of zones is in creating memory zone entity, that is a
      continuous chunk of memory available for allocation. If some chunk is
      not available, we simply do not put it in any zone.</para>
 
      <para>Zone is also serves for informational purposes, containing
      information about number of free and busy frames. Physical memory
      allocation is also done inside the certain zone. Allocation of zone
      frame must be organized by the <link linkend="frame_allocator">frame
      allocator</link> associated with the zone.</para>
 
      <para>Some of the architectures (mips32, ppc32) have only one zone, that
      covers whole physical memory, and the others (like ia32) may have
      multiple zones. Information about zones on current machine is stored in
      BIOS hardware tables or can be hardcoded into kernel during compile
      time.</para>
    </section>
 
    <section id="frame_allocator">
      <title>Frame allocator</title>
 
      <para><mediaobject id="frame_alloc">
          <imageobject role="html">
            <imagedata fileref="images/frame_alloc.png" format="PNG" />
          </imageobject>
 
          <imageobject role="fop">
            <imagedata fileref="images.vector/frame_alloc.svg" format="SVG" />
          </imageobject>
        </mediaobject></para>
 
      <formalpara>
        <title>Overview</title>
 
        <para>Frame allocator provides physical memory allocation for the
        kernel. Because of zonal organization of physical memory, frame
        allocator is always working in context of some zone, thus making
        impossible to allocate a piece of memory, which lays in different
        zone, which cannot happen, because two adjacent zones can be merged
        into one. Frame allocator is also being responsible to update
        information on the number of free/busy frames in zone. Physical memory
        allocation inside one <link linkend="zones_and_frames">memory
        zone</link> is being handled by an instance of <link
        linkend="buddy_allocator">buddy allocator</link> tailored to allocate
        blocks of physical memory frames.</para>
      </formalpara>
 
      <formalpara>
        <title>Allocation / deallocation</title>
 
        <para>Upon allocation request, frame allocator tries to find first
        zone, that can satisfy the incoming request (has required amount of
        free frames to allocate). During deallocation, frame allocator needs
        to find zone, that contain deallocated frame. This approach could
        bring up two potential problems: <itemizedlist>
            <listitem>
               Linear search of zones does not any good to performance, but number of zones is not expected to be high. And if yes, list of zones can be replaced with more time-efficient B-tree.
            </listitem>
 
            <listitem>
               Quickly find out if zone contains required number of frames to allocate and if this chunk of memory is properly aligned. This issue is perfectly solved bu the buddy allocator.
            </listitem>
          </itemizedlist></para>
      </formalpara>
    </section>
 
    <section id="buddy_allocator">
      <title>Buddy allocator</title>
 
      <section>
        <title>Overview</title>
 
        <para><mediaobject id="buddy_alloc">
            <imageobject role="html">
              <imagedata fileref="images/buddy_alloc.png" format="PNG" />
            </imageobject>
 
            <imageobject role="fop">
              <imagedata fileref="images.vector/buddy_alloc.svg" format="SVG" />
            </imageobject>
          </mediaobject></para>
 
        <para>In the buddy allocator, the memory is broken down into
        power-of-two sized naturally aligned blocks. These blocks are
        organized in an array of lists, in which the list with index i
        contains all unallocated blocks of size
        <mathphrase>2<superscript>i</superscript></mathphrase>. The index i is
        called the order of block. Should there be two adjacent equally sized
        blocks in the list i<mathphrase> </mathphrase>(i.e. buddies), the
        buddy allocator would coalesce them and put the resulting block in
        list <mathphrase>i + 1</mathphrase>, provided that the resulting block
        would be naturally aligned. Similarily, when the allocator is asked to
        allocate a block of size
        <mathphrase>2<superscript>i</superscript></mathphrase>, it first tries
        to satisfy the request from the list with index i. If the request
        cannot be satisfied (i.e. the list i is empty), the buddy allocator
        will try to allocate and split a larger block from the list with index
        i + 1. Both of these algorithms are recursive. The recursion ends
        either when there are no blocks to coalesce in the former case or when
        there are no blocks that can be split in the latter case.</para>
 
        <!--graphic fileref="images/mm1.png" format="EPS" /-->
 
        <para>This approach greatly reduces external fragmentation of memory
        and helps in allocating bigger continuous blocks of memory aligned to
        their size. On the other hand, the buddy allocator suffers increased
        internal fragmentation of memory and is not suitable for general
        kernel allocations. This purpose is better addressed by the <link
        linkend="slab">slab allocator</link>.</para>
      </section>
 
      <section>
        <title>Implementation</title>
 
        <para>The buddy allocator is, in fact, an abstract framework wich can
        be easily specialized to serve one particular task. It knows nothing
        about the nature of memory it helps to allocate. In order to beat the
        lack of this knowledge, the buddy allocator exports an interface that
        each of its clients is required to implement. When supplied with an
        implementation of this interface, the buddy allocator can use
        specialized external functions to find a buddy for a block, split and
        coalesce blocks, manipulate block order and mark blocks busy or
        available. For precise documentation of this interface, refer to
        <emphasis>"HelenOS Generic Kernel Reference Manual"</emphasis>.</para>
 
        <formalpara>
          <title>Data organization</title>
 
          <para>Each entity allocable by the buddy allocator is required to
          contain space for storing block order number and a link variable
          used to interconnect blocks within the same order.</para>
 
          <para>Whatever entities are allocated by the buddy allocator, the
          first entity within a block is used to represent the entire block.
          The first entity keeps the order of the whole block. Other entities
          within the block are assigned the magic value
          <constant>BUDDY_INNER_BLOCK</constant>. This is especially important
          for effective identification of buddies in a one-dimensional array
          because the entity that represents a potential buddy cannot be
          associated with <constant>BUDDY_INNER_BLOCK</constant> (i.e. if it
          is associated with <constant>BUDDY_INNER_BLOCK</constant> then it is
          not a buddy).</para>
 
          <para>The buddy allocator always uses the first frame to represent
          the frame block. This frame contains <varname>buddy_order</varname>
          variable to provide information about the block size it actually
          represents (
          <mathphrase>2<superscript>buddy_order</superscript></mathphrase>
          frames block). Other frames in block have this value set to magic
          <constant>BUDDY_INNER_BLOCK</constant> that is much greater than
          buddy <varname>max_order</varname> value.</para>
 
          <para>Each <varname>frame_t</varname> also contains pointer member
          to hold frame structure in the linked list inside one order.</para>
        </formalpara>
 
        <formalpara>
          <title>Allocation algorithm</title>
 
          <para>Upon <mathphrase>2<superscript>i</superscript></mathphrase>
          frames block allocation request, allocator checks if there are any
          blocks available at the order list <varname>i</varname>. If yes,
          removes block from order list and returns its address. If no,
          recursively allocates
          <mathphrase>2<superscript>i+1</superscript></mathphrase> frame
          block, splits it into two
          <mathphrase>2<superscript>i</superscript></mathphrase> frame blocks.
          Then adds one of the blocks to the <varname>i</varname> order list
          and returns address of another.</para>
        </formalpara>
 
        <formalpara>
          <title>Deallocation algorithm</title>
 
          <para>Check if block has so called buddy (another free
          <mathphrase>2<superscript>i</superscript></mathphrase> frame block
          that can be linked with freed block into the
          <mathphrase>2<superscript>i+1</superscript></mathphrase> block).
          Technically, buddy is a odd/even block for even/odd block
          respectively. Plus we can put an extra requirement, that resulting
          block must be aligned to its size. This requirement guarantees
          natural block alignment for the blocks coming out the allocation
          system.</para>
 
          <para>Using direct pointer arithmetics,
          <varname>frame_t::ref_count</varname> and
          <varname>frame_t::buddy_order</varname> variables, finding buddy is
          done at constant time.</para>
        </formalpara>
      </section>
    </section>
 
    <section id="slab">
      <title>Slab allocator</title>
 
      <section>
        <title>Overview</title>
 
        <para><termdef><glossterm>Slab</glossterm> represents a contiguous
        piece of memory, usually made of several physically contiguous
        pages.</termdef> <termdef><glossterm>Slab cache</glossterm> consists
        of one or more slabs.</termdef></para>
 
        <para>The majority of memory allocation requests in the kernel are for
        small, frequently used data structures. For this purpose the slab
        allocator is a perfect solution. The basic idea behind the slab
        allocator is to have lists of commonly used objects available packed
        into pages. This avoids the overhead of allocating and destroying
        commonly used types of objects such threads, virtual memory structures
        etc. Also due to the exact allocated size matching, slab allocation
        completely eliminates internal fragmentation issue.</para>
      </section>
 
      <section>
        <title>Implementation</title>
 
        <para><mediaobject id="slab_alloc">
            <imageobject role="html">
              <imagedata fileref="images/slab_alloc.png" format="PNG" />
            </imageobject>
 
            <imageobject role="fop">
              <imagedata fileref="images.vector/slab_alloc.svg" format="SVG" />
            </imageobject>
          </mediaobject></para>
 
        <para>The SLAB allocator is closely modelled after <ulink
        url="http://www.usenix.org/events/usenix01/full_papers/bonwick/bonwick_html/">
        OpenSolaris SLAB allocator by Jeff Bonwick and Jonathan Adams </ulink>
        with the following exceptions: <itemizedlist>
            <listitem>
               empty SLABS are deallocated immediately (in Linux they are kept in linked list, in Solaris ???)
            </listitem>
 
            <listitem>
               empty magazines are deallocated when not needed (in Solaris they are held in linked list in slab cache)
            </listitem>
          </itemizedlist> Following features are not currently supported but
        would be easy to do: <itemizedlist>
            <listitem>
               - cache coloring
            </listitem>
 
            <listitem>
               - dynamic magazine grow (different magazine sizes are already supported, but we would need to adjust allocation strategy)
            </listitem>
          </itemizedlist></para>
 
        <section>
          <title>Magazine layer</title>
 
          <para>Due to the extensive bottleneck on SMP architures, caused by
          global SLAB locking mechanism, making processing of all slab
          allocation requests serialized, a new layer was introduced to the
          classic slab allocator design. Slab allocator was extended to
          support per-CPU caches 'magazines' to achieve good SMP scaling.
          <termdef>Slab SMP perfromance bottleneck was resolved by introducing
          a per-CPU caching scheme called as <glossterm>magazine
          layer</glossterm></termdef>.</para>
 
          <para>Magazine is a N-element cache of objects, so each magazine can
          satisfy N allocations. Magazine behaves like a automatic weapon
          magazine (LIFO, stack), so the allocation/deallocation become simple
          push/pop pointer operation. Trick is that CPU does not access global
          slab allocator data during the allocation from its magazine, thus
          making possible parallel allocations between CPUs.</para>
 
          <para>Implementation also requires adding another feature as the
          CPU-bound magazine is actually a pair of magazines to avoid
          thrashing when during allocation/deallocatiion of 1 item at the
          magazine size boundary. LIFO order is enforced, which should avoid
          fragmentation as much as possible.</para>
 
          <para>Another important entity of magazine layer is a full magazine
          depot, that stores full magazines which are used by any of the CPU
          magazine caches to reload active CPU magazine. Magazine depot can be
          pre-filled with full magazines during initialization, but in current
          implementation it is filled during object deallocation, when CPU
          magazine becomes full.</para>
 
          <para>Slab allocator control structures are allocated from special
          slabs, that are marked by special flag, indicating that it should
          not be used for slab magazine layer. This is done to avoid possible
          infinite recursions and deadlock during conventional slab allocaiton
          requests.</para>
        </section>
 
        <section>
          <title>Allocation/deallocation</title>
 
          <para>Every cache contains list of full slabs and list of partialy
          full slabs. Empty slabs are immediately freed (thrashing will be
          avoided because of magazines).</para>
 
          <para>The SLAB allocator allocates lots of space and does not free
          it. When frame allocator fails to allocate the frame, it calls
          slab_reclaim(). It tries 'light reclaim' first, then brutal reclaim.
          The light reclaim releases slabs from cpu-shared magazine-list,
          until at least 1 slab is deallocated in each cache (this algorithm
          should probably change). The brutal reclaim removes all cached
          objects, even from CPU-bound magazines.</para>
 
          <formalpara>
            <title>Allocation</title>
 
            <para><emphasis>Step 1.</emphasis> When it comes to the allocation
            request, slab allocator first of all checks availability of memory
            in local CPU-bound magazine. If it is there, we would just "pop"
            the CPU magazine and return the pointer to object.</para>
 
            <para><emphasis>Step 2.</emphasis> If the CPU-bound magazine is
            empty, allocator will attempt to reload magazin, swapping it with
            second CPU magazine and returns to the first step.</para>
 
            <para><emphasis>Step 3.</emphasis> Now we are in the situation
            when both CPU-bound magazines are empty, which makes allocator to
            access shared full-magazines depot to reload CPU-bound magazines.
            If reload is succesful (meaning there are full magazines in depot)
            algoritm continues at Step 1.</para>
 
            <para><emphasis>Step 4.</emphasis> Final step of the allocation.
            In this step object is allocated from the conventional slab layer
            and pointer is returned.</para>
          </formalpara>
 
          <formalpara>
            <title>Deallocation</title>
 
            <para><emphasis>Step 1.</emphasis> During deallocation request,
            slab allocator will check if the local CPU-bound magazine is not
            full. In this case we will just push the pointer to this
            magazine.</para>
 
            <para><emphasis>Step 2.</emphasis> If the CPU-bound magazine is
            full, allocator will attempt to reload magazin, swapping it with
            second CPU magazine and returns to the first step.</para>
 
            <para><emphasis>Step 3.</emphasis> Now we are in the situation
            when both CPU-bound magazines are full, which makes allocator to
            access shared full-magazines depot to put one of the magazines to
            the depot and creating new empty magazine. Algoritm continues at
            Step 1.</para>
          </formalpara>
        </section>
      </section>
    </section>
 
    <!-- End of Physmem -->
  </section>
 
  <section>
    <title>Memory sharing</title>
 
    <para>Not implemented yet(?)</para>
  </section>
</chapter>
Subversion Repositories HelenOS-doc

(root)/design/trunk/src/ch_memory_management.xml – Rev 39 → 45
