Merge pull request #51 from redline6561/experimental

Release: 0.9.5!

NEWS.md

-## Changes for 0.9.5-dev (20xx):
+## Changes for 0.9.5 (2014-06-04):
 
-* A Twitter plugin to tweet your new posts. Thanks to @PuercoPop!
+* A plugin for Incremental builds, cutting runtime for generating
+  medium to large sites roughly in half!
+* A Twitter plugin to tweet about your new posts. Thanks to @PuercoPop!
 * Coleslaw now exports a `get-updated-files` function which can be
   used to get a list of file-status/file-name pairs that were changed
   in the last git push. There is also an exported `find-content-by-path`
-  function to retrieve content objects from the above file-name.
+  function to retrieve content objects from the above file-name. These
+  were used by both the Twitter and Incremental plugins.
+* The usual bugfixes, performance improvements, and documentation tweaks.
 
 ## Changes for 0.9.4 (2014-05-05):
 

README.md

 > drinking coffee, reading, writing, eating chips and salsa. I remember a gentleness
 > behind the enormous bushy eyebrows and that we called him Coleslaw. - anon
 
-Coleslaw aims to be flexible blog software suitable for replacing a single-user static site compiler such as Jekyll.
+Coleslaw aims to be flexible blog software suitable for replacing a single-user static site generator such as [Jekyll](http://jekyllrb.com/).
 
 ## Features
 * Git for storage
-* RSS and Atom feeds!
-* Markdown Support with Code Highlighting provided by [colorize](http://www.cliki.net/colorize).
+* RSS and Atom feeds
+* Markdown Support with Code Highlighting provided by [colorize](http://www.cliki.net/colorize)
   * Currently supports: Common Lisp, Emacs Lisp, Scheme, C, C++, Java, Python, Erlang, Haskell, Obj-C, Diff.
 
 * A [Plugin API](http://github.com/redline6561/coleslaw/blob/master/docs/plugin-api.md) and [**plugins**](http://github.com/redline6561/coleslaw/blob/master/docs/plugin-use.md) for...
   * Static Pages
+  * Sitemap generation
+  * Incremental builds
   * Analytics via Google
   * Comments via [Disqus](http://disqus.com/)
   * Hosting via [Github Pages](https://pages.github.com/), [Heroku](http://heroku.com/), or [Amazon S3](http://aws.amazon.com/s3/)
+  * [Tweeting](http://twitter.com/) about new posts
   * Using LaTeX via [Mathjax](http://mathjax.org/)
-  * Using ReStructured Text
+  * Writing posts in ReStructured Text
   * Importing posts from [Wordpress](http://wordpress.org/)
-  * Sitemap generation
 
 * There is also a [Heroku buildpack](https://github.com/jsmpereira/coleslaw-heroku) maintained by Jose Pereira.
-* Example sites: 
+
+## Example Sites
   * [redlinernotes](http://redlinernotes.com/blog/)
   * [kenan-bolukbasi.log](http://kenanb.com/)
   * [Nothing Really Matters](http://ironhead.xs4all.nl/)

coleslaw.asd

 (defsystem #:coleslaw
   :name "coleslaw"
   :description "Flexible Lisp Blogware"
-  :version "0.9.5-dev"
+  :version "0.9.5"
   :license "BSD"
   :author "Brit Butler <redline6561@gmail.com>"
   :pathname "src/"

docs/hacking.md

 would be worthwhile to see how well [cl-markdown][clmd] performs as
 a replacement if this becomes an issue for users though we would lose
 source highlighting from [colorize][clrz] and should also investigate
-[pygments][pyg] as a replacement.
+[pygments][pyg] as a replacement. Using the new [incremental][incf] plugin
+reduced runtime to 1.36 seconds, almost cutting it in half.
 
 ## Core Concepts
 
 All current Content Types and Indexes implement the protocol faithfully.
 It consists of 2 "class" methods, 2 instance methods, and an invariant.
 
-There are also 4 helper functions provided that should prove useful in
+There are also 5 helper functions provided that should prove useful in
 implementing new content types.
 
 
   unique. Such a hash collision represents content on the site being
   shadowed/overwritten. This should be used in your `discover` method.
 
+- `delete-document`: Remove a document from *coleslaw*'s in-memory
+  database. This is currently only used by the incremental compilation
+  plugin.
+
 - `write-document`: Write the document out to disk as HTML. It takes
   an optional template name and render-args to pass to the template.
   This should be used in your `publish` method.
 [clmd]: https://github.com/gwkkwg/cl-markdown
 [clrz]: https://github.com/redline6561/colorize
 [pyg]: http://pygments.org/
+[incf]: https://github.com/redline6561/coleslaw/blob/master/plugins/incremental.lisp

docs/plugin-use.md

 # General Use
 
 * Add a list with the plugin name and settings to the ```:plugins```
-  section of your [.coleslawrc][config_file]. Plugin settings are described below.
+  section of your [.coleslawrc][config_file]. Plugin settings are
+  described below.
 
-* Available plugins are listed below with usage descriptions and config examples.
+* Available plugins are listed below with usage descriptions and
+  config examples.
 
 ## Analytics via Google
 
-**Description**: Provides traffic analysis through [Google Analytics](http://www.google.com/analytics/).
+**Description**: Provides traffic analysis through
+  [Google Analytics](http://www.google.com/analytics/).
 
 **Example**: `(analytics :tracking-code "google-provided-unique-id")`
 
 ## Comments via Disqus
 
-**Description**: Provides comment support through [Disqus](http://www.disqus.com/).
+**Description**: Provides comment support through
+  [Disqus](http://www.disqus.com/).
 
 **Example**: `(disqus :shortname "disqus-provided-unique-id")`
 
 ## Hosting via Github Pages
 
-**Description**: Allows hosting with CNAMEs via [github-pages](http://pages.github.com/). Parses the host from the `:domain` section of your config by default. Pass in a string to override.
+**Description**: Allows hosting with CNAMEs via
+  [github-pages](http://pages.github.com/). Parses the host from the
+  `:domain` section of your config by default. Pass in a string to
+  override.
 
 **Example**: `(gh-pages :cname t)`
 
+## Incremental Builds
+
+**Description**: Primarily a performance enhancement. Caches the
+  content database between builds with
+  [cl-store][http://common-lisp.net/project/cl-store/] to avoid
+  parsing the whole git repo every time. May become default
+  functionality instead of a plugin at some point. Substantially
+  reduces runtime for medium to large sites.
+
+**Example**: `(incremental)`
+
+**Setup**:
+- You must run the `examples/dump_db.sh` script to generate a database dump
+  for your site before enabling the incremental plugin.
+
 ## LaTeX via Mathjax
 
-**Description**: Provides LaTeX support through [Mathjax](http://www.mathjax.org/) for posts tagged with "math" and indexes containing such posts. Any text enclosed in $$ will be rendered, for example, ```$$ \lambda \scriptstyle{f}. (\lambda x. (\scriptstyle{f} (x x)) \lambda x. (\scriptstyle{f} (x x))) $$```.
+**Description**: Provides LaTeX support through
+  [Mathjax](http://www.mathjax.org/) for posts tagged with "math" and
+  indexes containing such posts. Any text enclosed in $$ will be
+  rendered, for example, ```$$ \lambda \scriptstyle{f}. (\lambda
+  x. (\scriptstyle{f} (x x)) \lambda x. (\scriptstyle{f} (x x)))
+  $$```.
 
 **Example**: ```(mathjax)```
 
 **Options**:
 
-- `:force`, when non-nil, will force the inclusion of MathJax on all posts.  Default value is `nil`.
+- `:force`, when non-nil, will force the inclusion of MathJax on all
+  posts.  Default value is `nil`.
 
-- `:location` specifies the location of the `MathJax.js` file.  The default value is `"http://cdn.mathjax.org/mathjax/latest/MathJax.js"`.  This is useful if you have a local copy of MathJax and want to use that version.
+- `:location` specifies the location of the `MathJax.js` file.  The
+  default value is `"http://cdn.mathjax.org/mathjax/latest/MathJax.js"`.
+  This is useful if you have a local copy of MathJax and want to use that
+  version.
 
-- `:preset` allows the specification of the config parameter of `MathJax.js`.  The default value is `"TeX-AMS-MML_HTMLorMML"`.
+- `:preset` allows the specification of the config parameter of
+  `MathJax.js`.  The default value is `"TeX-AMS-MML_HTMLorMML"`.
 
-- `:config` is used as supplementary inline configuration to the `MathJax.Hub.Config ({ ... });`. It is unused by default.
+- `:config` is used as supplementary inline configuration to the
+  `MathJax.Hub.Config ({ ... });`. It is unused by default.
 
 ## ReStructuredText
 
-**Description**: Some people really like [ReStructuredText](http://docutils.sourceforge.net/rst.html). Who knows why? But it only took one method to add, so yeah! Just create a post with `format: rst` and the plugin will do the rest.
+**Description**: Some people really like
+  [ReStructuredText](http://docutils.sourceforge.net/rst.html). Who
+  knows why? But it only took one method to add, so yeah! Just create
+  a post with `format: rst` and the plugin will do the rest.
 
 **Example**: `(rst)`
 
 ## S3 Hosting
 
-**Description**: Allows hosting your blog entirely via [Amazon S3](http://aws.amazon.com/s3/). It is suggested you closely follow the relevant [AWS guide](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html) to get the DNS setup correctly. Your `:auth-file` should match that described in the [ZS3 docs](http://www.xach.com/lisp/zs3/#file-credentials).
+**Description**: Allows hosting your blog entirely via
+  [Amazon S3](http://aws.amazon.com/s3/). It is suggested you closely
+  follow the relevant
+  [AWS guide](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html)
+  to get the DNS setup correctly. Your `:auth-file` should match that
+  described in the
+  [ZS3 docs](http://www.xach.com/lisp/zs3/#file-credentials).
 
-**Example**: `(s3 :auth-file "/home/redline/.aws_creds" :bucket "blog.redlinernotes.com")`
+**Example**: `(s3 :auth-file "/home/redline/.aws_creds" :bucket
+  "blog.redlinernotes.com")`
 
 ## Sitemap generator
 
-**Description**: This plugin generates a sitemap.xml under the page root, which is useful if you want google to crawl your site.
+**Description**: This plugin generates a sitemap.xml under the page
+  root, which is useful if you want google to crawl your site.
 
 **Example**: `(sitemap)`
 
 ## Static Pages
 
-**Description**: This plugin allows you to add `.page` files to your repo, that will be rendered to static pages at a designated URL.
+**Description**: This plugin allows you to add `.page` files to your
+  repo, that will be rendered to static pages at a designated URL.
 
 **Example**: `(static-pages)`
 
-## Wordpress Importer
-
-**NOTE**: This plugin really should be rewritten to act as a standalone script. It is designed for one time use and using it through a site config is pretty silly.
-
-**Description**: Import blog posts from Wordpress using their export tool. Blog entries will be read from the XML and converted into .post files. Afterwards the XML file will be deleted to prevent reimporting. Optionally an `:output` argument may be supplied to the plugin. If provided, it should be a directory in which to store the .post files. Otherwise, the value of `:repo` in your .coleslawrc will be used.
-
-**Example**: `(import :filepath "/home/redline/redlinernotes-export.timestamp.xml" :output "/home/redlinernotes/blog/")`
-
-[config_file]: http://github.com/redline6561/coleslaw/blob/master/examples/single-site.coleslawrc
-
 ## Twitter
 
-**Description**: This plugin tweets every time a new post is added to your repo. See Setup for an example of how to get your access token & secret.
+**Description**: This plugin tweets every time a new post is added to
+  your repo. See Setup for an example of how to get your access token
+  & secret.
 
-**Example**: `(twitter :api-key "<api-key>" :api-secret "<api-seret" :access-token "<access-token>" :access-secret "<access-secret>")`
+**Example**: `(twitter :api-key "<api-key>"
+                       :api-secret "<api-secret>"
+                       :access-token "<access-token>"
+                       :access-secret "<access-secret>")`
 
 **Setup**:
 - Create a new [twitter app](https://apps.twitter.com/). Take note of the api key & secret.
 
 ;; Use the api key & secret to get a URL where a pin code will be handled to you.
 (chirp:initiate-authentication
- :api-key "D1pMCK17gI10bQ6orBPS0w"
- :api-secret "BfkvKNRRMoBPkEtDYAAOPW4s2G9U8Z7u3KAf0dBUA")
+  :api-key "D1pMCK17gI10bQ6orBPS0w"
+  :api-secret "BfkvKNRRMoBPkEtDYAAOPW4s2G9U8Z7u3KAf0dBUA")
 ;; => "https://api.twitter.com/oauth/authorize?oauth_token=cJIw9MJM5HEtQqZKahkj1cPn3m3kMb0BYEp6qhaRxfk"
 
 ;; Exchange the pin code for an access token and and access secret. Take note
 ;; => "18403733-bXtuum6qbab1O23ltUcwIk2w9NS3RusUFiuum4D3w"
 ;;    "zDFsFSaLerRz9PEXqhfB0h0FNfUIDgbEe59NIHpRWQbWk"
 
-;; Finally verify the credentials 
+;; Finally verify the credentials
 (chirp:account/verify-credentials)
 #<CHIRP-OBJECTS:USER PuercoPop #18405433>
 ```
+
+## Wordpress Importer
+
+**NOTE**: This plugin really should be rewritten to act as a
+  standalone script. It is designed for one time use and using it
+  through a site config is pretty silly.
+
+**Description**: Import blog posts from Wordpress using their export
+  tool. Blog entries will be read from the XML and converted into
+  .post files. Afterwards the XML file will be deleted to prevent
+  reimporting. Optionally an `:output` argument may be supplied to the
+  plugin. If provided, it should be a directory in which to store the
+  .post files. Otherwise, the value of `:repo` in your .coleslawrc
+  will be used.
+
+**Example**: `(import :filepath "/home/redline/redlinernotes-export.timestamp.xml"
+                      :output "/home/redlinernotes/blog/")`
+
+[config_file]: http://github.com/redline6561/coleslaw/blob/master/examples/example.coleslawrc

examples/dump-db.lisp

+(eval-when (:compile-toplevel :load-toplevel :execute)
+  (ql:quickload '(coleslaw cl-store)))
+
+(in-package :coleslaw)
+
+(defun main ()
+  (let ((db-file (rel-path (user-homedir-pathname) ".coleslaw.db")))
+    (format t "~%~%Coleslaw loaded. Attempting to load config file.~%")
+    (load-config "")
+    (format t "~%Config loaded. Attempting to load blog content.~%")
+    (load-content)
+    (format t "~%Content loaded. Attempting to dump content database.~%")
+    (cl-store:store *site* db-file)
+    (format t "~%Content database saved to ~s!~%~%" (namestring db-file))))
+
+(main)
+(exit)

examples/dump_db.sh

+#!/bin/sh
+
+LISP=sbcl
+
+## Disclaimer:
+## I have not tested that all lisps take the "--load" flag.
+## This code might spontaneously combust your whole everything.
+
+$LISP --load "dump-db.lisp"

plugins/incremental.lisp

+(eval-when (:compile-toplevel :load-toplevel)
+  (ql:quickload 'cl-store))
+
+(defpackage :coleslaw-incremental
+  (:use :cl)
+  (:import-from :alexandria #:when-let)
+  (:import-from :coleslaw #:*config*
+                          #:content
+                          #:index
+                          #:discover
+                          #:get-updated-files
+                          #:find-content-by-path
+                          #:add-document
+                          #:delete-document
+                          ;; Private
+                          #:all-subclasses
+                          #:do-subclasses
+                          #:read-content
+                          #:construct
+                          #:rel-path
+                          #:repo
+                          #:update-content-metadata)
+  (:export #:enable))
+
+(in-package :coleslaw-incremental)
+
+;; In contrast to the original incremental plans, full of shoving state into
+;; the right place by hand and avoiding writing pages to disk that hadn't
+;; changed, the new plan is to only avoid redundant parsing of content in
+;; the git repo. The rest of coleslaw's operation is "fast enough".
+;;
+;;   Prior to enabling the plugin a user must have a cl-store dump of the
+;;   database at ~/.coleslaw.db. There is a dump_db shell script in
+;;   examples to generate the database dump.
+;;
+;; We're gonna be a bit dirty here and monkey patch. The compilation model
+;; still isn't an "exposed" part of Coleslaw. After some experimentation maybe
+;; we'll settle on an interface.
+
+(defun coleslaw::load-content ()
+  (let ((db-file (rel-path (user-homedir-pathname) ".coleslaw.db")))
+    (setf coleslaw::*site* (cl-store:restore db-file))
+    (loop for (status path) in (get-updated-files)
+       for file-path = (rel-path (repo *config*) path)
+       do (update-content status file-path))
+    (update-content-metadata)
+    ;; Discover's :before method will delete any possibly outdated indexes.
+    (do-subclasses (itype index)
+      (discover itype))
+    (cl-store:store coleslaw::*site* db-file)))
+
+(defun update-content (status path)
+  (cond ((string= "D" status) (process-change :deleted path))
+        ((string= "M" status) (process-change :modified path))
+        ((string= "A" status) (process-change :added path))))
+
+(defgeneric process-change (status path &key &allow-other-keys)
+  (:documentation "Updates the database as needed for the STATUS change to PATH.")
+  (:method :around (status path &key)
+    (let ((extension (pathname-type path))
+          (ctypes (all-subclasses (find-class 'content))))
+      ;; This feels way too clever. I wish I could think of a better option.
+      (flet ((class-name-p (x class)
+               (string-equal x (symbol-name (class-name class)))))
+        ;; If the updated file's extension doesn't match one of our content types,
+        ;; we don't need to mess with it at all. Otherwise, since the class is
+        ;; annoyingly tricky to determine, pass it along.
+        (when-let (ctype (find extension ctypes :test #'class-name-p))
+          (call-next-method status path :ctype ctype))))))
+
+(defmethod process-change ((status (eql :deleted)) path &key)
+  (let ((old (find-content-by-path path)))
+    (delete-document old)))
+
+(defmethod process-change ((status (eql :modified)) path &key ctype)
+  (let ((old (find-content-by-path path))
+        (new (construct ctype (read-content path))))
+    (delete-document old)
+    (add-document new)))
+
+(defmethod process-change ((status (eql :added)) path &key ctype)
+  (let ((new (construct ctype (read-content path))))
+    (add-document new)))
+
+(defun enable ())

plugins/parallel.lisp

+(eval-when (:compile-toplevel :load-toplevel)
+  (ql:quickload 'lparallel))
+
+(defpackage :coleslaw-parallel
+  (:use :cl)
+  (:export #:enable))
+
+(in-package :coleslaw-parallel)
+
+;; TODO: The bulk of the speedup here should come from parallelizing discover.
+;; Publish will also benefit. Whether it's better to spin off threads for each
+;; content type/index type or the operations *within* discover/publish is not
+;; known, the higher granularity of doing it at the iterating over types level
+;; is certainly easier to prototype though.
+
+(defun enable ())

src/documents.lisp

         (error "There is already an existing document with the url ~a" url)
         (setf (gethash url *site*) document))))
 
+(defun delete-document (document)
+  "Given a DOCUMENT, delete it from the in-memory database."
+  (remhash (page-url document) *site*))
+
 (defun write-document (document &optional theme-fn &rest render-args)
   "Write the given DOCUMENT to disk as HTML. If THEME-FN is present,
 use it as the template passing any RENDER-ARGS."

src/packages.lisp

            #:get-updated-files
            #:theme-fn
            ;; The Document Protocol
-           #:add-document
-           #:find-all
-           #:purge-all
            #:discover
            #:publish
            #:page-url
            #:render
+           #:find-all
+           #:purge-all
+           #:add-document
+           #:delete-document
            #:write-document))

src/util.lisp

   "Create an instance of CLASS-NAME with the given ARGS."
   (apply 'make-instance class-name args))
 
+;; Thanks to bknr-web for this bit of code.
+(defun all-subclasses (class)
+  "Return a list of all the subclasses of CLASS."
+  (let ((subclasses (closer-mop:class-direct-subclasses class)))
+    (append subclasses (loop for subclass in subclasses
+                          nconc (all-subclasses subclass)))))
+
 (defmacro do-subclasses ((var class) &body body)
   "Iterate over the subclasses of CLASS performing BODY with VAR
 lexically bound to the current subclass."
-  (alexandria:with-gensyms (klasses all-subclasses)
-    `(labels ((,all-subclasses (class)
-                (let ((subclasses (closer-mop:class-direct-subclasses class)))
-                  (append subclasses (loop for subclass in subclasses
-                                        nconc (,all-subclasses subclass))))))
-       (let ((,klasses (,all-subclasses (find-class ',class))))
-         (loop for ,var in ,klasses do ,@body)))))
+  (alexandria:with-gensyms (klasses)
+    `(let ((,klasses (all-subclasses (find-class ',class))))
+       (loop for ,var in ,klasses do ,@body))))
 
 (defmacro do-files ((var path &optional extension) &body body)
   "For each file under PATH, run BODY. If EXTENSION is provided, only run