david
/
xowa
mirror of https://github.com/gnosygnu/xowa.git


			
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271
							<!DOCTYPE html>
<html dir="ltr">
<head>
  <meta http-equiv="content-type" content="text/html;charset=UTF-8" />
  <title>Dev/Design/Sqlite/File-sizes - XOWA</title>
  <link rel="shortcut icon" href="https://gnosygnu.github.io/xowa/xowa_logo.png" />
  <link rel="stylesheet" href="https://gnosygnu.github.io/xowa/xowa_common.css" type="text/css">
  
</head>
<body class="mediawiki ltr sitedir-ltr ns-0 ns-subject skin-vector action-submit vector-animateLayout" spellcheck="false">
<div id="mw-page-base" class="noprint"></div>
<div id="mw-head-base" class="noprint"></div>
<div id="content" class="mw-body">
  <h1 id="firstHeading" class="firstHeading"><span>Dev/Design/Sqlite/File-sizes</span></h1>
  <div id="bodyContent" class="mw-body-content">
    <div id="siteSub">From XOWA: the free, open-source, offline wiki application</div>
    <div id="contentSub"></div>
    <div id="mw-content-text" lang="en" dir="ltr" class="mw-content-ltr">
    
    <p>
      The XOWA sqlite import currently defaults to a multi-file format. This format is chosen for two reasons:
    </p>
    <ul>
      <li>
        <b>Large wikis and FAT32</b>: 
        <ul>
          <li>
            Most flash memory cards use a <a href="http://en.wikipedia.org/wiki/FAT32" rel="nofollow" class="external text">FAT32</a> file-system. FAT32 is particulary convenient when exchanging files between Windows, Linux, Mac OS X and Android.
          </li>
          <li>
            FAT32 has a limit of 4GB for any one file. A large wiki like en.wikipedia.org will easily take 20 GB.
          </li>
          <li>
            Multiple files allow the 20 GB data to be broken into smaller pieces: each less than 4 GB
          </li>
        </ul>
      </li>
    </ul>
    <ul>
      <li>
        <b>Slight performance gains</b>
      </li>
    </ul>
    <dl>
      <dd>
        A smaller database file may be easier to query than a large one because all the pages will be grouped closer together on disk
      </dd>
      <dd>
        For example, consider a wiki page that requires 50 template pages. 
        <dl>
          <dd>
            With a single-file format, these 50 pages may be scattered anywhere over the 20 GB file.
          </dd>
          <dd>
            With a multi-file format, these 50 pages may be scattered anywhere over a smaller 280 MB file. A disk drive will have to seek over a smaller section of disk. For a smaller wiki, the entire template file may be stored in the hard disk cache.
          </dd>
        </dl>
      </dd>
    </dl>
    <p>
      The file format is controlled by other arguments
    </p>
    <div id="toc" class="toc">
      <div id="toctitle" class="toctitle">
        <h2>
          Contents
        </h2>
      </div>
      <ul>
        <li class="toclevel-1 tocsection-1">
          <a href="#ns_file_map"><span class="tocnumber">1</span> <span class="toctext">ns_file_map</span></a>
        </li>
        <li class="toclevel-1 tocsection-2">
          <a href="#db_text_max_value"><span class="tocnumber">2</span> <span class="toctext">db_text_max value</span></a>
        </li>
        <li class="toclevel-1 tocsection-3">
          <a href="#db_categorylink_max_and_db_wikidata_max_value"><span class="tocnumber">3</span> <span class="toctext">db_categorylink_max and db_wikidata_max value</span></a>
        </li>
        <li class="toclevel-1 tocsection-4">
          <a href="#db_wikidata_max_value"><span class="tocnumber">4</span> <span class="toctext">db_wikidata_max value</span></a>
        </li>
      </ul>
    </div>
    <h2>
      <span class="mw-headline" id="ns_file_map">ns_file_map</span>
    </h2>
    <p>
      The ns_file_map argument is a new-line/semi-colon delimited string. The default value is the following:
    </p>
<pre>
Template;Module
</pre>
    <p>
      Note that each line has a list of namespace names. Multiple namespaces can be delimited with the ";". The namespace name must be the "canonical" English name.
    </p>
    <p>
      Note that an empty string will default everything to be stored in the core database. If a single file database is desired, specify "".
    </p>
    <h2>
      <span class="mw-headline" id="db_text_max_value">db_text_max value</span>
    </h2>
    <p>
      This is a number that represents the maximum number of MB of text data that can be stored in the file. Note the following
    </p>
    <ul>
      <li>
        Once a file reaches that number, it will spill over into a new file.
      </li>
    </ul>
    <dl>
      <dd>
        For example, file 002 is the text database. After 3,000 MB of text data is stored in file 002, the next 3,000 MB of text data will be stored in file 003.
      </dd>
    </dl>
    <ul>
      <li>
        The number is a rough approximation of total database size. A precise value cannot be used b/c of the following non-deterministic variables: 
        <ul>
          <li>
            Sqlite database page size (data / indexes will not fill up an entire page)
          </li>
          <li>
            Sqlite table / database overhead
          </li>
        </ul>
      </li>
    </ul>
    <dl>
      <dd>
        As such, please use a number which is 80% of the desired size. For example, if you want a database no greater than 4,000 MB (4.0 GB), use 3,000
      </dd>
    </dl>
    <h2>
      <span class="mw-headline" id="db_categorylink_max_and_db_wikidata_max_value">db_categorylink_max and db_wikidata_max value</span>
    </h2>
    <p>
      This is a number that represents the maximum number of MB of categorylink data that can be stored in the file. Note the following:
    </p>
    <ul>
      <li>
        This number functions similarly to the db_text_max value above. (Once the max is reached, new data will spill over into a new file)
      </li>
      <li>
        However, it is more precise than db_text_max. The number specified is 90% of the actual value (presumably due to less page fragmentation)
      </li>
    </ul>
    <h2>
      <span class="mw-headline" id="db_wikidata_max_value">db_wikidata_max value</span>
    </h2>
    <p>
      This is a number that represents the maximum number of MB of wikidata label data that can be stored in the file. Note the following:
    </p>
    <ul>
      <li>
        This number only affects www.wikidata.org wikis
      </li>
      <li>
        This number only recognizes 0 and not 0. 
        <ul>
          <li>
            To put all wikidata data in one database, use 0
          </li>
          <li>
            To put all wikidata data in another database, use any number &gt; 0
          </li>
        </ul>
      </li>
    </ul>
  
    </div>
  </div>
</div>


<div id="mw-head" class="noprint">
  <div id="left-navigation">
    <div id="p-namespaces" class="vectorTabs">
      <h3>Namespaces</h3>
      <ul>
        <li  id="ca-nstab-main" class="selected"><span><a id="ca-nstab-main-href" href="index.html">Page</a></span></li>
      </ul>
    </div>
  </div>
</div>

<div id='mw-panel' class='noprint'>
  <div id='p-logo'>
    <a style="background-image: url(https://gnosygnu.github.io/xowa/xowa_logo.png);" href="http://xowa.org/" title="Visit the main page"></a>
  </div>
  <div class="portal" id='xowa-portal-home'>
    <h3>XOWA</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/index.html" title='Visit the main page'>Main page</a></li>
        <li><a href="http://xowa.org/screenshots.html" title='See screenshots of XOWA'>Screenshots</a></li>
        <li><a href="https://www.youtube.com/watch?v=q0qbXYXEH6M" title="See a video of XOWA Desktop in action">Video</a></li>
        <li><a href="http://xowa.org/home/wiki/Help/Download_XOWA.html" title='Download the XOWA application'>Download XOWA</a></li>
        <li><a href="http://xowa.org/home/wiki/Dashboard/Image_databases.html" title='Download offline wikis and image databases'>Download wikis</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-started'>
    <h3>Getting started</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/home/wiki/App/Setup/System_requirements.html" title='Get XOWA&apos;s system requirements'>Requirements</a></li>
        <li><a href="http://xowa.org/home/wiki/App/Setup/Installation.html" title='Get instructions for installing XOWA'>Installation</a></li>
        <li><a href="http://xowa.org/home/wiki/App/Import/Simple_Wikipedia.html" title='Learn how to set up Simple Wikipedia'>Simple Wikipedia</a></li>
        <li><a href="http://xowa.org/home/wiki/App/Import/English_Wikipedia.html" title='Learn how to set up English Wikipedia'>English Wikipedia</a></li>
        <li><a href="http://xowa.org/home/wiki/App/Import/Other_wikis.html" title='Learn how to set up other Wikipedias'>Other Wikipedias</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-android'>
    <h3>Android</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/home/wiki/Android/Setup.html" title='Setup XOWA on your Android device'>Setup</a></li>
        <li><a href="https://www.youtube.com/watch?v=jsMTBxGweUw" title="See a video of XOWA Android in action">Video</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-help'>
    <h3>Help</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/home/wiki/Help/About.html" title='Get more information about XOWA'>About</a></li>
        <li><a href="http://xowa.org/home/wiki/Help/Contents.html" title='View a list of help topics'>Contents</a></li>
        <li><a href="http://xowa.org/home/wiki/Help/Media.html" title='Read what others have written about XOWA'>Media</a></li>
        <li><a href="http://xowa.org/home/wiki/Help/Feedback.html" title='Questions? Comments? Leave feedback for XOWA'>Feedback</a></li>
      </ul>
    </div>
  </div>
  
  <div class="portal" id='xowa-portal-blog'>
    <h3>Blog</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/home/wiki/Blog.html" title='Follow XOWA''s development process'>Current</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-links'>
    <h3>Links</h3>
    <div class="body">
      <ul>
        <li><a href="http://dumps.wikimedia.org/backup-index.html" title="Get wiki datababase dumps directly from Wikimedia">Wikimedia dumps</a></li>
        <li><a href="https://archive.org/search.php?query=xowa" title="Search archive.org for XOWA files">XOWA @ archive.org</a></li>
        <li><a href="http://en.wikipedia.org" title="Visit Wikipedia (and compare to XOWA!)">English Wikipedia</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-donate'>
    <h3>Donate</h3>
    <div class="body">
      <ul>
        <li><a href="https://archive.org/donate/index.php" title="Support archive.org!">archive.org</a></li><!-- listed first due to recent fire damages: http://blog.archive.org/2013/11/06/scanning-center-fire-please-help-rebuild/ -->
        <li><a href="https://donate.wikimedia.org/wiki/Special:FundraiserRedirector" title="Support Wikipedia!">Wikipedia</a></li>
        <li><a href="http://xowa.org/home/wiki/Help/Donate.html" title="Support XOWA!">XOWA</a></li>
      </ul>
    </div>
  </div>
  
</div>
</body>
</html>