david
/
xowa
mirror of https://github.com/gnosygnu/xowa.git


			
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259
							<!DOCTYPE html>
<html dir="ltr">
<head>
  <meta http-equiv="content-type" content="text/html;charset=UTF-8" />
  <title>App/Category - XOWA</title>
  <link rel="shortcut icon" href="https://gnosygnu.github.io/xowa/xowa_logo.png" />
  <link rel="stylesheet" href="https://gnosygnu.github.io/xowa/xowa_common.css" type="text/css">
  
</head>
<body class="mediawiki ltr sitedir-ltr ns-0 ns-subject skin-vector action-submit vector-animateLayout" spellcheck="false">
<div id="mw-page-base" class="noprint"></div>
<div id="mw-head-base" class="noprint"></div>
<div id="content" class="mw-body">
  <h1 id="firstHeading" class="firstHeading"><span>App/Category</span></h1>
  <div id="bodyContent" class="mw-body-content">
    <div id="siteSub">From XOWA: the free, open-source, offline wiki application</div>
    <div id="contentSub"></div>
    <div id="mw-content-text" lang="en" dir="ltr" class="mw-content-ltr">
    
    <p>
      As of v3.9.2.1 XOWA has one Category system: v3.
    </p>
    <ul>
      <li>
        The first part of this page will discuss v3.
      </li>
      <li>
        The second part of the page is an archived copy of the earlier v1 and v2 explanation
      </li>
    </ul>
    <h2>
      <span class="mw-headline" id="Version_3">Version 3</span>
    </h2>
    <p>
      Version 3 was introduced to handle Categories for HTML dumps on PC and Android. The high-level details are as follows:
    </p>
    <ul>
      <li>
        <b>Uses the Wikimedia categorylinks and page_props dumps</b>: Like v2, v3 downloads separate MediaWiki dumps for categorylinks.sql and page_props.sql . Both files are needed to generate accurate renditions of the Wikipedia Category system.
      </li>
      <li>
        <b>Generates "*-xtn.category.*.xowa" files</b>: v3 stores all the category info in "*-xtn.category.*.xowa" files. For smaller wikis, the data is stored instead in the "-core.xowa" file
      </li>
      <li>
        <b>Works with both Wikitext databases and HTML databases</b>: v3 will work with wikis imported by "Import/Online" (Wikitext) as well as "Download Central" (HTML)
      </li>
      <li>
        <b>Is backwards compatible with v1 and v2</b>: v3.9.2.1 will work with v1, v2, and v3 category systems
      </li>
      <li>
        <b>Is generated automatically with import</b>: v3 is now generated automatically when importing a wiki. Previously, v2 would require a separate post-processing step under Import/Offline.
      </li>
      <li>
        <b>Smaller size</b>: v3 makes some database changes to reduce file size. For English Wikipedia, that means a difference between 10 GB and 8 GB. 8 GB may sound like a lot for Categories, but keep in mind there are over 100 million page to category links.
      </li>
      <li>
        <b>Does not work for text database dumps</b>: XOWA originally started off storing files in text files instead of sqlite files. I switched over to SQLite three years ago and phased out text databases two years ago. It's possible that some users with old wikis (3 years old) may still have these text databases. If so, then the new Category system won't work.
      </li>
    </ul>
    <hr>
    <h2>
      <span class="mw-headline" id="Version_1">Version 1</span>
    </h2>
    <p>
      Version 1 is a simplistic category system.
    </p>
    <ul>
      <li>
        It relies only on page content inside the xml file. It does not use any of the category*.sql dumps.
      </li>
    </ul>
    <p>
      Note the following limitations:
    </p>
    <ul>
      <li>
        Does not work with large categories. It gets linearly worse with more members (do not use it to load a category with over 10,000 members)
      </li>
      <li>
        Does not support paging. If a category has 1,000 members, it will load title information on all 1,000 (instead of just the first 200)
      </li>
      <li>
        Does not use sortkey. For example, Jimmy Wales will alphabetize under J (for Jimmy Wales) instead of W (for Wales, Jimmy)
      </li>
      <li>
        Does not accurately reflect page membership in categories.
      </li>
    </ul>
    <dl>
      <dd>
        For example, most hidden categories are added to a template which is then included in a page.
      </dd>
      <dd>
        Specifically, a page called "File:GNU.png" may belong to "All free media". However, the "File:GNU.png" page doesn't have the [[Category:All_free_media]] but instead embeds a template {{All_non_free_media}} which has the [[Category:All_free_media]]
      </dd>
      <dd>
        Since a full parse (with templates) of the entire xml file would take many hours, this membership data is omitted.
      </dd>
    </dl>
    <p>
      V1 should be considered obsolete. No signficant changes will be made to it, as V2 is the official category system.
    </p>
    <p>
      However, because V1 is faster to setup than V2, it still remains the default (with a strong recommendation to upgrade to V2 when time permits)
    </p>
    <h2>
      <span class="mw-headline" id="Version_2">Version 2</span>
    </h2>
    <p>
      Version 2 is an accurate category system.
    </p>
    <ul>
      <li>
        It uses the Wikimedia dump files: categorylinks.sql, page_props.sql
      </li>
    </ul>
    <p>
      It addresses each of the limitations of version 1, including
    </p>
    <ul>
      <li>
        Works with large categories
      </li>
      <li>
        Supports paging
      </li>
      <li>
        Uses sortkey
      </li>
      <li>
        Accurately includes all members of a category
      </li>
    </ul>
    <p>
      It has a few limitations:
    </p>
    <ul>
      <li>
        It requires additional dump files (as mentioned above).
      </li>
      <li>
        It takes longer to setup. A separate .sql file must be parsed. For English Wikipedia this process takes about another hour.
      </li>
      <li>
        It takes more disk space. The v2 system stores sortkeys individually per entry (just like Wikipedia). However this text data greatly increases the overall file size. English Wikipedia will have about 10.0 GB of extra data.
      </li>
    </ul>
    <p>
      V2 is the official category system and should generate Category pages just like Wikipedia.
    </p>
    <p>
      For more information about V2 setup see <a href="/wiki/App/Category/Building" id="xolnki_2" title="App/Category/Building">App/Category/Building</a>
    </p>
    <p>
      For more information about V2 internals see <a href="/wiki/App/Category/Internals" id="xolnki_3" title="App/Category/Internals">App/Category/Internals</a>
    </p>
  
    </div>
  </div>
</div>


<div id="mw-head" class="noprint">
  <div id="left-navigation">
    <div id="p-namespaces" class="vectorTabs">
      <h3>Namespaces</h3>
      <ul>
        <li  id="ca-nstab-main" class="selected"><span><a id="ca-nstab-main-href" href="index.html">Page</a></span></li>
      </ul>
    </div>
  </div>
</div>

<div id='mw-panel' class='noprint'>
  <div id='p-logo'>
    <a style="background-image: url(https://gnosygnu.github.io/xowa/xowa_logo.png);" href="http://xowa.org/" title="Visit the main page"></a>
  </div>
  <div class="portal" id='xowa-portal-home'>
    <h3>XOWA</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/index.html" title='Visit the main page'>Main page</a></li>
        <li><a href="http://xowa.org/screenshots.html" title='See screenshots of XOWA'>Screenshots</a></li>
        <li><a href="https://www.youtube.com/watch?v=q0qbXYXEH6M" title="See a video of XOWA Desktop in action">Video</a></li>
        <li><a href="http://xowa.org/home/wiki/Help/Download_XOWA.html" title='Download the XOWA application'>Download XOWA</a></li>
        <li><a href="http://xowa.org/home/wiki/Dashboard/Image_databases.html" title='Download offline wikis and image databases'>Download wikis</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-started'>
    <h3>Getting started</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/home/wiki/App/Setup/System_requirements.html" title='Get XOWA&apos;s system requirements'>Requirements</a></li>
        <li><a href="http://xowa.org/home/wiki/App/Setup/Installation.html" title='Get instructions for installing XOWA'>Installation</a></li>
        <li><a href="http://xowa.org/home/wiki/App/Import/Simple_Wikipedia.html" title='Learn how to set up Simple Wikipedia'>Simple Wikipedia</a></li>
        <li><a href="http://xowa.org/home/wiki/App/Import/English_Wikipedia.html" title='Learn how to set up English Wikipedia'>English Wikipedia</a></li>
        <li><a href="http://xowa.org/home/wiki/App/Import/Other_wikis.html" title='Learn how to set up other Wikipedias'>Other Wikipedias</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-android'>
    <h3>Android</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/home/wiki/Android/Setup.html" title='Setup XOWA on your Android device'>Setup</a></li>
        <li><a href="https://www.youtube.com/watch?v=jsMTBxGweUw" title="See a video of XOWA Android in action">Video</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-help'>
    <h3>Help</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/home/wiki/Help/About.html" title='Get more information about XOWA'>About</a></li>
        <li><a href="http://xowa.org/home/wiki/Help/Contents.html" title='View a list of help topics'>Contents</a></li>
        <li><a href="http://xowa.org/home/wiki/Help/Media.html" title='Read what others have written about XOWA'>Media</a></li>
        <li><a href="http://xowa.org/home/wiki/Help/Feedback.html" title='Questions? Comments? Leave feedback for XOWA'>Feedback</a></li>
      </ul>
    </div>
  </div>
  
  <div class="portal" id='xowa-portal-blog'>
    <h3>Blog</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/home/wiki/Blog.html" title='Follow XOWA''s development process'>Current</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-links'>
    <h3>Links</h3>
    <div class="body">
      <ul>
        <li><a href="http://dumps.wikimedia.org/backup-index.html" title="Get wiki datababase dumps directly from Wikimedia">Wikimedia dumps</a></li>
        <li><a href="https://archive.org/search.php?query=xowa" title="Search archive.org for XOWA files">XOWA @ archive.org</a></li>
        <li><a href="http://en.wikipedia.org" title="Visit Wikipedia (and compare to XOWA!)">English Wikipedia</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-donate'>
    <h3>Donate</h3>
    <div class="body">
      <ul>
        <li><a href="https://archive.org/donate/index.php" title="Support archive.org!">archive.org</a></li><!-- listed first due to recent fire damages: http://blog.archive.org/2013/11/06/scanning-center-fire-please-help-rebuild/ -->
        <li><a href="https://donate.wikimedia.org/wiki/Special:FundraiserRedirector" title="Support Wikipedia!">Wikipedia</a></li>
        <li><a href="http://xowa.org/home/wiki/Help/Donate.html" title="Support XOWA!">XOWA</a></li>
      </ul>
    </div>
  </div>
  
</div>
</body>
</html>