david
/
xowa
mirror of https://github.com/gnosygnu/xowa.git


			
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359
							<!DOCTYPE html>
<html dir="ltr">
<head>
  <meta http-equiv="content-type" content="text/html;charset=UTF-8" />
  <title>App/Search/Score - XOWA</title>
  <link rel="shortcut icon" href="https://gnosygnu.github.io/xowa/xowa_logo.png" />
  <link rel="stylesheet" href="https://gnosygnu.github.io/xowa/xowa_common.css" type="text/css">
  
</head>
<body class="mediawiki ltr sitedir-ltr ns-0 ns-subject skin-vector action-submit vector-animateLayout" spellcheck="false">
<div id="mw-page-base" class="noprint"></div>
<div id="mw-head-base" class="noprint"></div>
<div id="content" class="mw-body">
  <h1 id="firstHeading" class="firstHeading"><span>App/Search/Score</span></h1>
  <div id="bodyContent" class="mw-body-content">
    <div id="siteSub">From XOWA: the free, open-source, offline wiki application</div>
    <div id="contentSub"></div>
    <div id="mw-content-text" lang="en" dir="ltr" class="mw-content-ltr">
    
    <p>
      XOWA calculates a score to every page for the purpose of ranking search results.
    </p>
    <div id="toc" class="toc">
      <div id="toctitle">
        <h2>
          Contents
        </h2>
      </div>
      <ul>
        <li class="toclevel-1 tocsection-1">
          <a href="#Overview"><span class="tocnumber">1</span> <span class="toctext">Overview</span></a>
        </li>
        <li class="toclevel-1 tocsection-2">
          <a href="#Scaling_.2F_Ranking"><span class="tocnumber">2</span> <span class="toctext">Scaling / Ranking</span></a> 
          <ul>
            <li class="toclevel-2 tocsection-3">
              <a href="#Scaling"><span class="tocnumber">2.1</span> <span class="toctext">Scaling</span></a>
            </li>
            <li class="toclevel-2 tocsection-4">
              <a href="#Ranking"><span class="tocnumber">2.2</span> <span class="toctext">Ranking</span></a>
            </li>
          </ul>
        </li>
        <li class="toclevel-1 tocsection-5">
          <a href="#Calculation"><span class="tocnumber">3</span> <span class="toctext">Calculation</span></a> 
          <ul>
            <li class="toclevel-2 tocsection-6">
              <a href="#PageRank"><span class="tocnumber">3.1</span> <span class="toctext">PageRank</span></a>
            </li>
            <li class="toclevel-2 tocsection-7">
              <a href="#Short_pages_are_penalized"><span class="tocnumber">3.2</span> <span class="toctext">Short pages are penalized</span></a>
            </li>
            <li class="toclevel-2 tocsection-8">
              <a href="#Scores_are_re-scaled"><span class="tocnumber">3.3</span> <span class="toctext">Scores are re-scaled</span></a>
            </li>
          </ul>
        </li>
      </ul>
    </div>
    <h2>
      <span class="mw-headline" id="Overview">Overview</span>
    </h2>
    <p>
      From a broad perspective, the following happens:
    </p>
    <ul>
      <li>
        A <a href="https://en.wikipedia.org/wiki/PageRank" rel="nofollow" class="external text">PageRank</a> score is calculated for a page. This score is <a href="https://en.wikipedia.org/wiki/Feature_scaling" rel="nofollow" class="external text">scaled</a> from 0 to 1,000,000
      </li>
      <li>
        A page-length score is then calculated for the page. This score is <a href="https://en.wikipedia.org/wiki/Percentile_rank" rel="nofollow" class="external text">ranked</a> from 0 to 1,000,000
      </li>
      <li>
        The PageRank score is then multiplied by a ratio if it has a low page-length score
      </li>
      <li>
        The resulting PageRank score is then ranked from 0 to 1,000,000.
      </li>
    </ul>
    <p>
      <br>
    </p>
    <h2>
      <span class="mw-headline" id="Scaling_.2F_Ranking">Scaling / Ranking</span>
    </h2>
    <p>
      XOWA uses "scaling" and "ranking" at various stages to calculate the score.
    </p>
    <h3>
      <span class="mw-headline" id="Scaling">Scaling</span>
    </h3>
    <p>
      A simplified definition of scaling is converting a number from one range to another range based on proportion. For a more thorough definition, see <a href="https://en.wikipedia.org/wiki/Feature_scaling" rel="nofollow" class="external text">the Wikipedia page on feature scaling</a>
    </p>
    <p>
      For example, let's say you have a score of 100 in a range of 0 to 400 and want to scale it to 0 to 1000. The following steps would be involved:
    </p>
    <ul>
      <li>
        Take 100 and divide it by 400. This yields .25
      </li>
      <li>
        Take .25 and multiply it by 1000. This yields 250.
      </li>
    </ul>
    <p>
      The following formula is the basis for scaling:
    </p>
    <p>
      <span id='xowa_math_txt_0'>newScore = \frac{oldScore - \text{min}(oldRange)} {\text{max}(oldRange)-\text{min}(oldRange)} \cdot (\text{max}(newRange)-\text{min}(newRange))</span>
    </p>
    <p>
      Or, to use the example from above:
    </p>
    <p>
      <span id='xowa_math_txt_0'>250 = \frac{100 - 0} {400-0} \cdot (1000-0)</span>
    </p>
    <h3>
      <span class="mw-headline" id="Ranking">Ranking</span>
    </h3>
    <p>
      A simplified definition of ranking is assigning a number based on its order in a population of numbers. For those familiar with a school setting, this is "grading on a curve". For a more thorough definition, see <a href="https://en.wikipedia.org/wiki/Percentile_rank" rel="nofollow" class="external text">the Wikipedia page on percentile ranks</a>
    </p>
    <p>
      For example, let's say you have the following:
    </p>
    <ul>
      <li>
        A minimum score of 0
      </li>
      <li>
        A maximum score of 100
      </li>
      <li>
        5 entities with the following scores 
        <ul>
          <li>
            A : 99
          </li>
          <li>
            B : 10
          </li>
          <li>
            C : 42
          </li>
          <li>
            D : 71
          </li>
          <li>
            E : 56
          </li>
        </ul>
      </li>
    </ul>
    <p>
      Ranking would do the following:
    </p>
    <ul>
      <li>
        Sort the scores 
        <ul>
          <li>
            A : 99
          </li>
          <li>
            D : 71
          </li>
          <li>
            E : 56
          </li>
          <li>
            C : 42
          </li>
          <li>
            B : 10
          </li>
        </ul>
      </li>
      <li>
        Calculate the "interval" for each score by taking the maximum and dividing by the number of scores. 
        <ul>
          <li>
            In this case, this would be 20: 100 / 5
          </li>
        </ul>
      </li>
      <li>
        Assign each score a new score based on its proportionate position in the sort. 
        <ul>
          <li>
            A : 100
          </li>
          <li>
            D : 80
          </li>
          <li>
            E : 60
          </li>
          <li>
            C : 40
          </li>
          <li>
            B : 20
          </li>
        </ul>
      </li>
    </ul>
    <p>
      <br>
    </p>
    <h2>
      <span class="mw-headline" id="Calculation">Calculation</span>
    </h2>
    <h3>
      <span class="mw-headline" id="PageRank">PageRank</span>
    </h3>
    <p>
      The basis of XOWA's page score is <a href="https://en.wikipedia.org/wiki/PageRank" rel="nofollow" class="external text">PageRank</a>.
    </p>
    <p>
      In brief, PageRank will give high scores to pages which are:
    </p>
    <ol>
      <li>
        linked to by many pages
      </li>
      <li>
        linked to by pages which have a high score.
      </li>
    </ol>
    <p>
      Note that #2 is recursive (a page will have a high score only if it is linked to by many pages). For more info, a good starting point is <a href="https://en.wikipedia.org/wiki/PageRank" rel="nofollow" class="external text">the Wikipedia page on PageRank</a>.
    </p>
    <p>
      After XOWA calculates the PageRank, XOWA then scales this score in a range of 0 to 1,000,000
    </p>
    <h3>
      <span class="mw-headline" id="Short_pages_are_penalized">Short pages are penalized</span>
    </h3>
    <p>
      XOWA penalizes short pages. This reduces the effect of small stub pages which are linked to by many articles, but mostly from boilerplate navigation boxes.
    </p>
    <p>
      XOWA ranks pages based on page-length. The generated score is in a range from 0 to 1,000,000
    </p>
    <p>
      Currently the method is:
    </p>
    <ul>
      <li>
        If the page is in the bottom 60% of page lengths...
      </li>
      <li>
        Then multiply the page score by that percentage.
      </li>
    </ul>
    <p>
      For example, a page that has a length in the bottom 10% and a score of 1000, will have a score of 100 (1000 * 10%). In contrast, a page that has a length in the top 65% with a score of 9,000 will still have a score of 9,000.
    </p>
    <p>
      Note that this calculation is an ad-hoc creation and will probably change in the future.
    </p>
    <h3>
      <span class="mw-headline" id="Scores_are_re-scaled">Scores are re-scaled</span>
    </h3>
    <p>
      The final step is to take the modified score and rank it from 0 to 1,000,000. Note that this final score is an integer (not a decimal / float)
    </p>
  
    </div>
  </div>
</div>


<div id="mw-head" class="noprint">
  <div id="left-navigation">
    <div id="p-namespaces" class="vectorTabs">
      <h3>Namespaces</h3>
      <ul>
        <li  id="ca-nstab-main" class="selected"><span><a id="ca-nstab-main-href" href="index.html">Page</a></span></li>
      </ul>
    </div>
  </div>
</div>

<div id='mw-panel' class='noprint'>
  <div id='p-logo'>
    <a style="background-image: url(https://gnosygnu.github.io/xowa/xowa_logo.png);" href="http://xowa.org/" title="Visit the main page"></a>
  </div>
  <div class="portal" id='xowa-portal-home'>
    <h3>XOWA</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/index.html" title='Visit the main page'>Main page</a></li>
        <li><a href="http://xowa.org/screenshots.html" title='See screenshots of XOWA'>Screenshots</a></li>
        <li><a href="http://xowa.org/wiki/home/page/Help/Download_XOWA.html" title='Download the XOWA application'>Download XOWA</a></li>
        <li><a href="http://xowa.org/wiki/home/page/Dashboard/Image_databases.html" title='Download offline wikis and image databases'>Download wikis</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-stargin'>
    <h3>Getting started</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/wiki/home/page/App/Setup/System_requirements.html" title='Get XOWA&apos;s system requirements'>Requirements</a></li>
        <li><a href="http://xowa.org/wiki/home/page/App/Setup/Installation.html" title='Get instructions for installing XOWA'>Installation</a></li>
        <li><a href="http://xowa.org/wiki/home/page/App/Import/Simple_Wikipedia.html" title='Learn how to set up Simple Wikipedia'>Simple Wikipedia</a></li>
        <li><a href="http://xowa.org/wiki/home/page/App/Import/English_Wikipedia.html" title='Learn how to set up English Wikipedia'>English Wikipedia</a></li>
        <li><a href="http://xowa.org/wiki/home/page/App/Import/Other_wikis.html" title='Learn how to set up other Wikipedias'>Other Wikipedias</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-help'>
    <h3>Help</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/wiki/home/page/Help/About.html" title='Get more information about XOWA'>About</a></li>
        <li><a href="http://xowa.org/wiki/home/page/Help/Contents.html" title='View a list of help topics'>Contents</a></li>
        <li><a href="http://xowa.org/wiki/home/page/Help/Media.html" title='Read what others have written about XOWA'>Media</a></li>
        <li><a href="http://xowa.org/wiki/home/page/Help/Feedback.html" title='Questions? Comments? Leave feedback for XOWA'>Feedback</a></li>
      </ul>
    </div>
  </div>
  
  <div class="portal" id='xowa-portal-blog'>
    <h3>Blog</h3>
    <div class="body">
      <ul>
        <li><a href="http://xowa.org/wiki/home/page/Blog.html" title='Follow XOWA''s development process'>Current</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-links'>
    <h3>Links</h3>
    <div class="body">
      <ul>
        <li><a href="http://dumps.wikimedia.org/backup-index.html" title="Get wiki datababase dumps directly from Wikimedia">Wikimedia dumps</a></li>
        <li><a href="https://archive.org/search.php?query=xowa" title="Search archive.org for XOWA files">XOWA @ archive.org</a></li>
        <li><a href="http://en.wikipedia.org" title="Visit Wikipedia (and compare to XOWA!)">English Wikipedia</a></li>
      </ul>
    </div>
  </div>

  <div class="portal" id='xowa-portal-donate'>
    <h3>Donate</h3>
    <div class="body">
      <ul>
        <li><a href="https://archive.org/donate/index.php" title="Support archive.org!">archive.org</a></li><!-- listed first due to recent fire damages: http://blog.archive.org/2013/11/06/scanning-center-fire-please-help-rebuild/ -->
        <li><a href="https://donate.wikimedia.org/wiki/Special:FundraiserRedirector" title="Support Wikipedia!">Wikipedia</a></li>
        <!-- <li><a href="" title="Support XOWA! (but only after you've supported archive.org and Wikipedia)">XOWA</a></li> -->
      </ul>
    </div>
  </div>
</div>
</body>
</html>