TaviPatches/DiskCache

---

This patch adds disk caching functionality to Tavi 0.26. All the view-action pages are cached to disk and served from there whenever possible, so as to skip database read and wiki code parsing and processing. You can find the patch at the bottom of this page. --CiprianPopovici

What the patch modifies

• It adds foreign_code/cache/lib.php, which contains the cache support, and includes it from other files.
  • Edit function cache_die() and customize the explanatory banners to your liking. If you wish, you can move the texts to global config variables or to the I18N support.
  • Please note that in case the page is successfully retrieved from cache, you will have the CSS stylesheets available!
• It adds several cache-related config variables to lib/defaults.php.
  • You'll have to copy these variables to lib/config.php and customize them
  • The default values disable the cache support!
  • Make sure you set a directory that the webserver user is allowed to write to!
• It reorganizes lib/init.php and lib/main.php by reordering the require() calls.
  • This is only needed if you want the cache system to be able to server as database failure fallback. If you don't, then delete changes to lib/db.php, lib/main.php and lib/init.php from the patch.
  • Why is the reordering necessary: lib/db.php needs the $page name; lib/pagestore.php uses lib/db.php; and in the official 'Tavi, lib/pagestore.php is used before $page is properly filtered and checked. So I've twisted things around so that $page is verified and primed as soon as possible, making it available to lib/db.php. Actually, I'd advise this reordering for the official 'Tavi as well; none of the $page checks in lib/main.php are database dependent.
• A few files are modified so they make use of the cache system: lib/view.php, lib/save.php, lib/db.php.

How the disk cache system works

• Every time someone saves a page (that is, at the end of action/save.php), the cached file for that page is deleted. This is the signal that the cache needs to be regenerated for that page.
• When someone views a page (that is, at the beginning of action/view.php), the system checks if a cached version is available. If so, it dumps the file contents to the browser and bails out without retrieving the page code from the database and without processing it.
  • For pages that are marked to be skipped in the preferences, the cache is never used from action/view.php.
  • How to tell if a page was served from cache or from the database: function cache_use_page() ads a timestamp HTML comment to the end, check out the page source.
• After someone has viewed a page (that is, at the end of action/view.php and only if we haven't bailed out as described above), the output buffer is also saved to disk as the cache for that page.
  • The buffer is saved only if the cached file is missing; which happens if action/save.php deleted it (ie. the page changed) or if the page is new and this is the first visit.
  • Pages marked to be skipped by cache also generate cache copies, they just don't use them. They will still need them for database failure fallback, see below.
• Furthermore, lib/db.php now calls cache_die() instead of die(), which will attempt to serve a cached version as well as output a banner explaining what is happening.
  • Note that in case of database failure, the cache system serves cached pages even for pages marked to be skipped. I believe this is sensible, since a cached copy is better than nothing.

How the disk cache is organized

The cached copies are written to a directory that needs to be read/write accessable by the user that the webserver runs as. The file names are MD5 hashes of the page names, so you don't have to worry about strange characters in the page names not making valid filesystem names.

A cache copy is simply deleted by a save and regenerated by the subsequent view of the page. Successive views won't regenerate the copy needlessly. A copy is recreated only if it's missing.

The size of the cache can run around at least a couple of MB for the average site, by "average" meaning with a decent amount of content, kind of like this 'Tavi site.

Remember to skip dynamic pages

You have to mark down at least some pages to be skipped by the cache system. Most obvious ones are the likes of RecentChanges, and in general any pages that make use of macros which are expanded in relation to global wiki changes.

Carefully consider whether you have other kind of dynamic content on your wiki pages. If you have some exotic PHP-generated random content, it will clash with the static cached copy functionality.

Discussion

The system created by this patch could use some improvements: --CiprianPopovici

• Improvement: Pages with macros that depend on other pages (like RecentChanges) could be cached too, instead of being skipped completely. Just make another array, call it CacheDynamicPages?, and add code that will delete the pages in it every time any page saves a new version. Regeneration will occur the next time someone views a dynamic page. It will stay cached as long as there are no saves anywhere. Not a dramatic improvement but it's better than none.
• Bug: The patch in this form doesn't care about page versions. If you view a version in the past it still shows the cache, which contains the latest version. Modify to either not use the cache for past versions, or make separate cache copies for each version. The former is probably best; 99% of the time people look at the latest version.
• Shortcoming: Cached copies never go away. Add code that will delete from cache when a page expires completely.
• Comment: In function cache_use_page of file cache/lib.php I think ob_start still must run if the cache file not exists: -- MiGi

  if (file_exists($file_name)) {
     ...
  }else{
     while (ob_get_length()) ob_end_clean();
     ob_start();
  }

• Moreover, my site uses a sidebar feature which trascludes other pages to show some information. In so doing caching the whole page is almost helpless for me. It will be more useful if it cache rather parseEngline outcome than the whole page. -- MiGi

The patch

---

diff -ruN tavi/action/save.php tavi.cache/action/save.php
--- tavi/action/save.php	2005-03-29 21:27:44.000000000 +0300
+++ tavi.cache/action/save.php	2005-09-13 10:30:21.000000000 +0300
@@ -1,6 +1,7 @@
 <?php
 // $Id: save.php,v 1.11 2005/03/29 18:27:44 holroy Exp $
 
+require_once('foreign_code/cache/lib.php');
 require(TemplateDir . '/save.php');
 require('lib/category.php');
 require('parse/save.php');
@@ -91,5 +92,7 @@
 
   $pagestore->unlock();                 // End "transaction".
 
+  // delete cached version so the next view will know it needs refreshing
+  @cache_delete_page($page);
 }
 ?>
diff -ruN tavi/action/view.php tavi.cache/action/view.php
--- tavi/action/view.php	2002-01-07 18:28:32.000000000 +0200
+++ tavi.cache/action/view.php	2005-09-13 10:30:46.000000000 +0300
@@ -1,6 +1,7 @@
 <?php
 // $Id: view.php,v 1.7 2002/01/07 16:28:32 smoonen Exp $
 
+require_once('foreign_code/cache/lib.php');
 require('parse/main.php');
 require('parse/macros.php');
 require('parse/html.php');
@@ -12,6 +13,9 @@
 {
   global $page, $pagestore, $ParseEngine, $version;
 
+  // attempt to use cached version, if applicable
+  @cache_use_page($page);
+
   $pg = $pagestore->page($page);
   if($version != '')
     { $pg->version = $version; }
@@ -25,5 +29,8 @@
                       'timestamp' => $pg->time,
                       'archive'   => $version != '',
                       'version'   => $pg->version));
+
+  // attempt to save cache, if necessary
+  @cache_save_page($page);
 }
 ?>
diff -ruN tavi/foreign_code/cache/lib.php tavi.cache/foreign_code/cache/lib.php
--- tavi/foreign_code/cache/lib.php	1970-01-01 02:00:00.000000000 +0200
+++ tavi.cache/foreign_code/cache/lib.php	2005-09-13 10:52:59.000000000 +0300
@@ -0,0 +1,123 @@
+<?php
+/********************************************************************************
+cache v1 Sep 2005, (c)Ciprian Popovici, ciprian zuavra net
+
+This program is free software; you can redistribute it and/or
+modify it under the terms of the GNU General Public License
+as published by the Free Software Foundation; either version 2
+of the License, or (at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details:
+<http://www.gnu.org/licenses/gpl.txt>
+********************************************************************************/
+
+function cache_die($error_string) {
+  global $page, $action;
+  global $CacheEnabled, $CacheIsDatabaseFallback;
+
+  if (!$CacheEnabled || !$CacheIsDatabaseFallback || $action!='view') {
+    echo $error_string;
+    exit;
+  }
+
+  echo '<div class="cache-attempt">Website database is down. Will attempt to fetch a static page version from cache instead.</div>';
+  $contents = cache_get_page($page);
+  if (!$contents) {
+    echo '<div class="cache-result">I could not find a cached version for this page. Sorry. Please go back and try other pages.</div>';
+  }
+  else {
+    echo '<div class="cache-result">You are now viewing cached pages. You will be unable to use any dynamic functionality (search, preferences, edit, history, diff), and dynamic pages (ex. WikiChanges) won\'t be updated.</div>';
+    echo '<div class="cache-contents">';
+    echo $contents;
+    echo '</div>';
+  }
+  exit;
+}
+
+function cache_skip_page($page) {
+  global $CacheEnabled, $CacheSkipPages;
+  
+  if (!$CacheEnabled) return true;
+  
+  if (is_array($CacheSkipPages)) {
+    $skipped = $CacheSkipPages;
+  }
+  elseif (is_string($CacheSkipPages)) {
+    $skipped = explode(',',$CacheSkipPages);
+  }
+  else return false;
+  
+  foreach ($skipped as $key => $value) {
+    $skipped[$key] = trim($value,'/');
+  }
+  
+  return in_array($page,$skipped);
+}
+
+function cache_use_page($page) {
+  global $CacheEnabled, $CacheDirectory;
+  
+  if (!$CacheEnabled) return true;
+  
+  if (cache_skip_page($page)) return false;
+
+  $file_name = $CacheDirectory.'/'.strtolower(md5($page));
+  if (file_exists($file_name)) {
+    while (ob_get_length()) ob_end_clean();
+    ob_start('ob_gzhandler');
+    cache_dump_page($page);
+    echo "\n<!-- cache from ".date("F d Y H:i:s T", filemtime($file_name))." -->\n";
+    exit;
+  }
+}
+
+function cache_delete_page($page) {
+  global $CacheEnabled, $CacheDirectory;
+  
+  if (!$CacheEnabled) return true;
+
+  $file_name = $CacheDirectory.'/'.strtolower(md5($page));
+  if (file_exists($file_name)) {
+    return @unlink($file_name);
+  }
+}
+
+function cache_save_page($page) {
+  global $CacheEnabled, $CacheDirectory;
+  
+  if (!$CacheEnabled) return true;
+
+  $file_name = $CacheDirectory.'/'.strtolower(md5($page));
+  if (!file_exists($file_name)) {
+    $f = fopen($file_name,'w');
+    if (!$f) return false;
+    $wrote = fwrite($f, ob_get_contents());
+    fclose($f);
+    if ($wrote != ob_get_length()) {
+      @unlink($file_name);
+    }
+  }
+}
+
+function cache_dump_page($page) {
+  global $CacheDirectory;
+
+  $file_name = $CacheDirectory.'/'.strtolower(md5($page));
+  if (file_exists($file_name)) {
+    return @readfile($file_name);
+  }
+}
+
+function cache_get_page($page) {
+  global $CacheDirectory;
+
+  $file_name = $CacheDirectory.'/'.strtolower(md5($page));
+  if (file_exists($file_name)) {
+    return @file_get_contents($file_name);
+  }
+}
+
+?>
\ No newline at end of file
diff -ruN tavi/lib/db.php tavi.cache/lib/db.php
--- tavi/lib/db.php	2003-12-16 20:07:46.000000000 +0200
+++ tavi.cache/lib/db.php	2005-09-13 10:31:55.000000000 +0300
@@ -3,22 +3,26 @@
 
 // MySQL database abstractor.  It should be easy to port this to other
 //   databases, such as PostgreSQL.
+require_once('foreign_code/cache/lib.php');
+
 class WikiDB
 {
   var $handle;
 
   function WikiDB($persistent, $server, $user, $pass, $database)
   {
+    global $page;
+
     if($persistent)
       { $this->handle = mysql_pconnect($server, $user, $pass); }
     else
       { $this->handle = mysql_connect($server, $user, $pass); }
 
     if($this->handle <= 0)
-      { die(LIB_ErrorDatabaseConnect); }
+      { cache_die(LIB_ErrorDatabaseConnect); }
 
     if(mysql_select_db($database, $this->handle) == false)
-      { die(LIB_ErrorDatabaseSelect); }
+      { cache_die(LIB_ErrorDatabaseSelect); }
   }
 
   function query($text)
diff -ruN tavi/lib/defaults.php tavi.cache/lib/defaults.php
--- tavi/lib/defaults.php	2005-03-30 19:31:48.000000000 +0300
+++ tavi.cache/lib/defaults.php	2005-09-13 10:33:51.000000000 +0300
@@ -14,6 +14,17 @@
 // value.  This will override the default set here.
 //**********************************************************************
 
+// whether to use page cache
+$CacheEnabled = false;
+// what directory to use for storing cached pages
+$CacheDirectory = $_SERVER["DOCUMENT_ROOT"];
+// if db connection fails, try to fall back to cached versions?
+// this only works if the cache is enabled and the cache dir is valid
+$CacheIsDatabaseFallback = false;
+// comma-separated list of pages that should not be cached (watch those extra spaces!)
+// this can also be an array of strings; the code will use it either way
+$CacheSkipPages = "";
+
 // The following variables establish the format for WikiNames in this wiki.
 $UpperPtn = "[A-Z\xc0-\xde]";
 $LowerPtn = "[a-z\xdf-\xfe]";
diff -ruN tavi/lib/init.php tavi.cache/lib/init.php
--- tavi/lib/init.php	2005-03-30 23:05:49.000000000 +0300
+++ tavi.cache/lib/init.php	2005-09-13 10:52:00.000000000 +0300
@@ -2,12 +2,8 @@
 // $Id: init.php,v 1.18 2005/03/30 20:05:49 holroy Exp $
 
 // General initialization code.
-
-require('lib/defaults.php');
-require('config.php');
-require('lib/url.php');
-require('lib/pagestore.php');
-require('lib/rate.php');
+require_once('lib/pagestore.php');
+require_once('lib/rate.php');
 
 $PgTbl = $DBTablePrefix . 'pages';
 $IwTbl = $DBTablePrefix . 'interwiki';
@@ -52,10 +48,10 @@
 
 // Choose a textual language for this wiki
 if (defined('LANGUAGE_CODE')) {
-  require('lang/lang_'. LANGUAGE_CODE . '.php');
+  require_once('lang/lang_'. LANGUAGE_CODE . '.php');
 }
 // Due to definition of setConst, this will add those not defined yet
-require('lang/default.php');
+require_once('lang/default.php');
 
 if(!empty($prefstr))
 {
diff -ruN tavi/lib/main.php tavi.cache/lib/main.php
--- tavi/lib/main.php	2005-03-30 17:38:22.000000000 +0300
+++ tavi.cache/lib/main.php	2005-09-13 10:51:44.000000000 +0300
@@ -72,8 +72,11 @@
 $referrer     = isset($HTTP_POST_VARS['referrer'])
                 ? $HTTP_POST_VARS['referrer'] : '';
 
-require('lib/init.php');
-require('parse/transforms.php');
+// General initialization code.
+require_once('lib/defaults.php');
+require_once('config.php');
+require_once('lib/url.php');
+require_once('parse/transforms.php');
 
 // To add an action=x behavior, add an entry to this array.  First column
 //   is the file to load, second is the function to call, and third is how
@@ -105,6 +108,10 @@
 if(!validate_page($page))
   { die(LIB_ErrorInvalidPage); }
 
+// aditional settings and initialization
+require_once('foreign_code/cache/lib.php');
+require_once('lib/init.php');
+
 // Don't let people do too many things too quickly.
 if($ActionList[$action][2] != '')
   { rateCheck($pagestore->dbh, $ActionList[$action][2]); }

---

---

Document last modified Mon, 05 Dec 2005 09:00:02

Search: