Automated merge with ssh://bitbucket.org/DasIch/sphinx-version-tracking

CHANGES.jacobmason

@@ -0,0 +1,27 @@
+May 30: Added files builders/websupport.py, writers/websupport.py,
+websupport/api.py, and websupport/document.api. Provides a rudimentary 
+method of building websupport data, and rendering it as html.
+
+May 31-June 10: Continued changing way web support data is represented
+and accessed.
+
+June 14 - June 17: Continued making improvements to the web support package 
+and demo web application. Included sidebars, navlinks etc...
+
+June 21 - June 26: Implement server side search with two search adapters,
+one for Xapian and one for Whoosh
+
+June 28 - July 12: Implement voting system on the backend, and created a
+jQuery script to handle voting on the frontend.
+
+July 13 - July 19: Added documentation for the web support package.
+
+July 20 - July 27: Added a system to allow user's to propose changes to
+documentation along with comments.
+
+July 28 - August 3: Added tests for the web support package. Refactored
+sqlalchemy storage to be more efficient.
+
+August 4 - August 7: Added comment moderation system. Added more 
+documentation. General code cleanup.
+

doc/contents.rst

@@ -17,6 +17,7 @@ Sphinx documentation contents
    theming
    templating
    extensions
+   websupport
 
    faq
    glossary

doc/web/api.rst

@@ -0,0 +1,66 @@
+.. _websupportapi:
+
+.. currentmodule:: sphinx.websupport
+
+The WebSupport Class
+====================
+
+.. class:: WebSupport
+
+    The main API class for the web support package. All interactions
+    with the web support package should occur through this class.
+
+    The class takes the following keyword arguments:
+
+    srcdir
+        The directory containing reStructuredText source files.
+
+    builddir
+        The directory that build data and static files should be placed in.
+        This should be used when creating a :class:`WebSupport` object that
+        will be used to build data.
+
+    datadir:
+        The directory that the web support data is in. This should be used
+        when creating a :class:`WebSupport` object that will be used to
+        retrieve data.
+
+    search:
+        This may contain either a string (e.g. 'xapian') referencing a
+        built-in search adapter to use, or an instance of a subclass of
+        :class:`~sphinx.websupport.search.BaseSearch`.
+
+    storage:
+        This may contain either a string representing a database uri, or an
+        instance of a subclass of
+        :class:`~sphinx.websupport.storage.StorageBackend`. If this is not
+        provided a new sqlite database will be created.
+
+    moderation_callback:
+        A callable to be called when a new comment is added that is not
+        displayed. It must accept one argument: a dict representing the
+        comment that was added.
+
+    staticdir:
+        If static files are served from a location besides "/static", this
+        should be a string with the name of that location
+        (e.g. '/static_files').
+
+    docroot:
+        If the documentation is not served from the base path of a URL, this
+        should be a string specifying that path (e.g. 'docs')
+
+Methods
+~~~~~~~
+
+.. automethod:: sphinx.websupport.WebSupport.build
+
+.. automethod:: sphinx.websupport.WebSupport.get_document
+
+.. automethod:: sphinx.websupport.WebSupport.get_data
+
+.. automethod:: sphinx.websupport.WebSupport.add_comment
+
+.. automethod:: sphinx.websupport.WebSupport.process_vote
+
+.. automethod:: sphinx.websupport.WebSupport.get_search_results

doc/web/quickstart.rst

@@ -0,0 +1,268 @@
+.. _websupportquickstart:
+
+Web Support Quick Start
+=======================
+
+Building Documentation Data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To make use of the web support package in your application you'll
+need to build the data it uses. This data includes pickle files representing
+documents, search indices, and node data that is used to track where
+comments and other things are in a document. To do this you will need
+to create an instance of the :class:`~sphinx.websupport.WebSupport`
+class and call it's :meth:`~sphinx.websupport.WebSupport.build` method::
+
+    from sphinx.websupport import WebSupport
+
+    support = WebSupport(srcdir='/path/to/rst/sources/',
+                         builddir='/path/to/build/outdir',
+                         search='xapian')
+
+    support.build()
+
+This will read reStructuredText sources from `srcdir` and place the
+necessary data in `builddir`. The `builddir` will contain two
+sub-directories. One named "data" that contains all the data needed
+to display documents, search through documents, and add comments to
+documents. The other directory will be called "static" and contains static
+files that should be served from "/static".
+
+.. note::
+
+    If you wish to serve static files from a path other than "/static", you
+    can do so by providing the *staticdir* keyword argument when creating
+    the :class:`~sphinx.websupport.api.WebSupport` object.
+
+Integrating Sphinx Documents Into Your Webapp
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now that the data is built, it's time to do something useful with it.
+Start off by creating a :class:`~sphinx.websupport.WebSupport` object
+for your application::
+
+    from sphinx.websupport import WebSupport
+
+    support = WebSupport(datadir='/path/to/the/data',
+                         search='xapian')
+
+You'll only need one of these for each set of documentation you will be
+working with. You can then call it's
+:meth:`~sphinx.websupport.WebSupport.get_document` method to access
+individual documents::
+
+    contents = support.get_document('contents')
+
+This will return a dictionary containing the following items:
+
+* **body**: The main body of the document as HTML
+* **sidebar**: The sidebar of the document as HTML
+* **relbar**: A div containing links to related documents
+* **title**: The title of the document
+* **css**: Links to css files used by Sphinx
+* **js**: Javascript containing comment options
+
+This dict can then be used as context for templates. The goal is to be
+easy to integrate with your existing templating system. An example using
+`Jinja2 <http://jinja.pocoo.org/2/>`_ is:
+
+.. sourcecode:: html+jinja
+
+    {%- extends "layout.html" %}
+
+    {%- block title %}
+        {{ document.title }}
+    {%- endblock %}
+
+    {% block css %}
+        {{ super() }}
+        {{ document.css|safe }}
+        <link rel="stylesheet" href="/static/websupport-custom.css" type="text/css">
+    {% endblock %}
+
+    {%- block js %}
+        {{ super() }}
+        {{ document.js|safe }}
+    {%- endblock %}
+
+    {%- block relbar %}
+        {{ document.relbar|safe }}
+    {%- endblock %}
+
+    {%- block body %}
+        {{ document.body|safe }}
+    {%- endblock %}
+
+    {%- block sidebar %}
+        {{ document.sidebar|safe }}
+    {%- endblock %}
+
+Authentication
+--------------
+
+To use certain features such as voting it must be possible to authenticate
+users. The details of the authentication are left to your application.
+Once a user has been authenticated you can pass the user's details to certain
+:class:`~sphinx.websupport.WebSupport` methods using the *username* and
+*moderator* keyword arguments. The web support package will store the
+username with comments and votes. The only caveat is that if you allow users
+to change their username you must update the websupport package's data::
+
+    support.update_username(old_username, new_username)
+
+*username* should be a unique string which identifies a user, and *moderator*
+should be a boolean representing whether the user has moderation
+privilieges. The default value for *moderator* is *False*.
+
+An example `Flask <http://flask.pocoo.org/>`_ function that checks whether
+a user is logged in and then retrieves a document is::
+
+    from sphinx.websupport.errors import *
+
+    @app.route('/<path:docname>')
+    def doc(docname):
+        username = g.user.name if g.user else ''
+        moderator = g.user.moderator if g.user else False
+        try:
+            document = support.get_document(docname, username, moderator)
+        except DocumentNotFoundError:
+            abort(404)
+        return render_template('doc.html', document=document)
+
+The first thing to notice is that the *docname* is just the request path.
+This makes accessing the correct document easy from a single view.
+If the user is authenticated then the username and moderation status are
+passed along with the docname to
+:meth:`~sphinx.websupport.WebSupport.get_document`. The web support package
+will then add this data to the COMMENT_OPTIONS that are used in the template.
+
+.. note::
+
+    This only works works if your documentation is served from your
+    document root. If it is served from another directory, you will
+    need to prefix the url route with that directory, and give the `docroot`
+    keyword argument when creating the web support object::
+
+        support = WebSupport(...
+                             docroot='docs')
+
+        @app.route('/docs/<path:docname>')
+
+Performing Searches
+~~~~~~~~~~~~~~~~~~~
+
+To use the search form built-in to the Sphinx sidebar, create a function
+to handle requests to the url 'search' relative to the documentation root.
+The user's search query will be in the GET parameters, with the key `q`.
+Then use the :meth:`~sphinx.websupport.WebSupport.get_search_results` method
+to retrieve search results. In `Flask <http://flask.pocoo.org/>`_ that
+would be like this::
+
+    @app.route('/search')
+    def search():
+        q = request.args.get('q')
+        document = support.get_search_results(q)
+        return render_template('doc.html', document=document)
+
+Note that we used the same template to render our search results as we
+did to render our documents. That's because
+:meth:`~sphinx.websupport.WebSupport.get_search_results` returns a context
+dict in the same format that
+:meth:`~sphinx.websupport.WebSupport.get_document` does.
+
+Comments & Proposals
+~~~~~~~~~~~~~~~~~~~~
+
+Now that this is done it's time to define the functions that handle
+the AJAX calls from the script. You will need three functions. The first
+function is used to add a new comment, and will call the web support method
+:meth:`~sphinx.websupport.WebSupport.add_comment`::
+
+    @app.route('/docs/add_comment', methods=['POST'])
+    def add_comment():
+        parent_id = request.form.get('parent', '')
+        node_id = request.form.get('node', '')
+        text = request.form.get('text', '')
+        proposal = request.form.get('proposal', '')
+        username = g.user.name if g.user is not None else 'Anonymous'
+        comment = support.add_comment(text, node_id='node_id',
+                                      parent_id='parent_id',
+                                      username=username, proposal=proposal)
+        return jsonify(comment=comment)
+
+You'll notice that both a `parent_id` and `node_id` are sent with the
+request. If the comment is being attached directly to a node, `parent_id`
+will be empty. If the comment is a child of another comment, then `node_id`
+will be empty. Then next function handles the retrieval of comments for a
+specific node, and is aptly named
+:meth:`~sphinx.websupport.WebSupport.get_data`::
+
+    @app.route('/docs/get_comments')
+    def get_comments():
+        username = g.user.name if g.user else None
+        moderator = g.user.moderator if g.user else False
+        node_id = request.args.get('node', '')
+        data = support.get_data(parent_id, user_id)
+        return jsonify(**data)
+
+The final function that is needed will call
+:meth:`~sphinx.websupport.WebSupport.process_vote`, and will handle user
+votes on comments::
+
+    @app.route('/docs/process_vote', methods=['POST'])
+    def process_vote():
+        if g.user is None:
+            abort(401)
+        comment_id = request.form.get('comment_id')
+        value = request.form.get('value')
+        if value is None or comment_id is None:
+            abort(400)
+        support.process_vote(comment_id, g.user.id, value)
+        return "success"
+
+Comment Moderation
+~~~~~~~~~~~~~~~~~~
+
+By default all comments added through
+:meth:`~sphinx.websupport.WebSupport.add_comment` are automatically
+displayed. If you wish to have some form of moderation, you can pass
+the `displayed` keyword argument::
+
+    comment = support.add_comment(text, node_id='node_id',
+                                  parent_id='parent_id',
+                                  username=username, proposal=proposal,
+                                  displayed=False)
+
+You can then create two new views to handle the moderation of comments. The
+first will be called when a moderator decides a comment should be accepted
+and displayed::
+
+    @app.route('/docs/accept_comment', methods=['POST'])
+    def accept_comment():
+        moderator = g.user.moderator if g.user else False
+        comment_id = request.form.get('id')
+        support.accept_comment(comment_id, moderator=moderator)
+        return 'OK'
+
+The next is very similar, but used when rejecting a comment::
+
+    @app.route('/docs/reject_comment', methods=['POST'])
+    def reject_comment():
+        moderator = g.user.moderator if g.user else False
+        comment_id = request.form.get('id')
+        support.reject_comment(comment_id, moderator=moderator)
+        return 'OK'
+
+To perform a custom action (such as emailing a moderator) when a new comment
+is added but not displayed, you can pass callable to the
+:class:`~sphinx.websupport.WebSupport` class when instantiating your support
+object::
+
+    def moderation_callback(comment):
+        Do something...
+
+    support = WebSupport(...
+                         moderation_callback=moderation_callback)
+
+The moderation callback must take one argument, which will be the same
+comment dict that is returned by add_comment.

doc/web/searchadapters.rst

@@ -0,0 +1,46 @@
+.. _searchadapters:
+
+.. currentmodule:: sphinx.websupport.search
+
+Search Adapters
+===============
+
+To create a custom search adapter you will need to subclass the
+:class:`~BaseSearch` class. Then create an instance of the new class
+and pass that as the `search` keyword argument when you create the
+:class:`~sphinx.websupport.WebSupport` object::
+
+    support = Websupport(srcdir=srcdir,
+                         builddir=builddir,
+                         search=MySearch())
+
+For more information about creating a custom search adapter, please see
+the documentation of the :class:`BaseSearch` class below.
+
+.. class:: BaseSearch
+
+    Defines an interface for search adapters.
+
+BaseSearch Methods
+~~~~~~~~~~~~~~~~~~
+
+    The following methods are defined in the BaseSearch class. Some methods
+    do not need to be overridden, but some (
+    :meth:`~sphinx.websupport.search.BaseSearch.add_document` and
+    :meth:`~sphinx.websupport.search.BaseSearch.handle_query`) must be
+    overridden in your subclass. For a working example, look at the
+    built-in adapter for whoosh.
+
+.. automethod:: sphinx.websupport.search.BaseSearch.init_indexing
+
+.. automethod:: sphinx.websupport.search.BaseSearch.finish_indexing
+
+.. automethod:: sphinx.websupport.search.BaseSearch.feed
+
+.. automethod:: sphinx.websupport.search.BaseSearch.add_document
+
+.. automethod:: sphinx.websupport.search.BaseSearch.query
+
+.. automethod:: sphinx.websupport.search.BaseSearch.handle_query
+
+.. automethod:: sphinx.websupport.search.BaseSearch.extract_context

doc/web/storagebackends.rst

@@ -0,0 +1,45 @@
+.. _storagebackends:
+
+.. currentmodule:: sphinx.websupport.storage
+
+Storage Backends
+================
+
+To create a custom storage backend you will need to subclass the
+:class:`~StorageBackend` class. Then create an instance of the new class
+and pass that as the `storage` keyword argument when you create the
+:class:`~sphinx.websupport.WebSupport` object::
+
+    support = Websupport(srcdir=srcdir,
+                         builddir=builddir,
+                         storage=MyStorage())
+
+For more information about creating a custom storage backend, please see
+the documentation of the :class:`StorageBackend` class below.
+
+.. class:: StorageBackend
+
+    Defines an interface for storage backends.
+
+StorageBackend Methods
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. automethod:: sphinx.websupport.storage.StorageBackend.pre_build
+
+.. automethod:: sphinx.websupport.storage.StorageBackend.add_node
+
+.. automethod:: sphinx.websupport.storage.StorageBackend.post_build
+
+.. automethod:: sphinx.websupport.storage.StorageBackend.add_comment
+
+.. automethod:: sphinx.websupport.storage.StorageBackend.delete_comment
+
+.. automethod:: sphinx.websupport.storage.StorageBackend.get_data
+
+.. automethod:: sphinx.websupport.storage.StorageBackend.process_vote
+
+.. automethod:: sphinx.websupport.storage.StorageBackend.update_username
+
+.. automethod:: sphinx.websupport.storage.StorageBackend.accept_comment
+
+.. automethod:: sphinx.websupport.storage.StorageBackend.reject_comment

doc/websupport.rst

@@ -0,0 +1,15 @@
+.. _websupport:
+
+Sphinx Web Support
+==================
+
+Sphinx provides a way to easily integrate Sphinx documentation
+into your web application. To learn more read the
+:ref:`websupportquickstart`.
+
+.. toctree::
+
+    web/quickstart
+    web/api
+    web/searchadapters
+    web/storagebackends

sphinx/builders/__init__.py

@@ -329,4 +329,5 @@ BUILTIN_BUILDERS = {
     'man':        ('manpage', 'ManualPageBuilder'),
     'changes':    ('changes', 'ChangesBuilder'),
     'linkcheck':  ('linkcheck', 'CheckExternalLinksBuilder'),
+    'websupport': ('websupport', 'WebSupportBuilder'),
 }

sphinx/builders/websupport.py

@@ -0,0 +1,150 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.builders.websupport
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    Builder for the web support package.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+import cPickle as pickle
+from os import path
+from cgi import escape
+import posixpath
+import shutil
+from docutils.io import StringOutput
+
+from sphinx.util.osutil import os_path, relative_uri, ensuredir, copyfile
+from sphinx.builders.html import StandaloneHTMLBuilder
+from sphinx.writers.websupport import WebSupportTranslator
+
+class WebSupportBuilder(StandaloneHTMLBuilder):
+    """
+    Builds documents for the web support package.
+    """
+    name = 'websupport'
+    out_suffix = '.fpickle'
+
+    def init_translator_class(self):
+        self.translator_class = WebSupportTranslator
+
+    def write_doc(self, docname, doctree):
+        destination = StringOutput(encoding='utf-8')
+        doctree.settings = self.docsettings
+
+        self.cur_docname = docname
+        self.secnumbers = self.env.toc_secnumbers.get(docname, {})
+        self.imgpath = '/' + posixpath.join(self.app.staticdir, '_images')
+        self.post_process_images(doctree)
+        self.dlpath = '/' + posixpath.join(self.app.staticdir, '_downloads')
+        self.docwriter.write(doctree, destination)
+        self.docwriter.assemble_parts()
+        body = self.docwriter.parts['fragment']
+        metatags = self.docwriter.clean_meta
+
+        ctx = self.get_doc_context(docname, body, metatags)
+        self.index_page(docname, doctree, ctx.get('title', ''))
+        self.handle_page(docname, ctx, event_arg=doctree)
+
+    def get_target_uri(self, docname, typ=None):
+        return docname
+
+    def load_indexer(self, docnames):
+        self.indexer = self.app.search
+        self.indexer.init_indexing(changed=docnames)
+
+    def handle_page(self, pagename, addctx, templatename='page.html',
+                    outfilename=None, event_arg=None):
+        # This is mostly copied from StandaloneHTMLBuilder. However, instead
+        # of rendering the template and saving the html, create a context
+        # dict and pickle it.
+        ctx = self.globalcontext.copy()
+        ctx['pagename'] = pagename
+
+        def pathto(otheruri, resource=False,
+                   baseuri=self.get_target_uri(pagename)):
+            if not resource:
+                otheruri = self.get_target_uri(otheruri)
+                return relative_uri(baseuri, otheruri) or '#'
+            else:
+                return '/' + posixpath.join(self.app.staticdir, otheruri)
+        ctx['pathto'] = pathto
+        ctx['hasdoc'] = lambda name: name in self.env.all_docs
+        ctx['encoding'] = encoding = self.config.html_output_encoding
+        ctx['toctree'] = lambda **kw: self._get_local_toctree(pagename, **kw)
+        self.add_sidebars(pagename, ctx)
+        ctx.update(addctx)
+
+        self.app.emit('html-page-context', pagename, templatename,
+                      ctx, event_arg)
+
+        # Create a dict that will be pickled and used by webapps.
+        css = '<link rel="stylesheet" href="%s" type=text/css />' % \
+            pathto('_static/pygment.css', 1)
+        doc_ctx = {'body': ctx.get('body', ''),
+                   'title': ctx.get('title', ''),
+                   'css': css,
+                   'js': self._make_js(ctx)}
+        # Partially render the html template to proved a more useful ctx.
+        template = self.templates.environment.get_template(templatename)
+        template_module = template.make_module(ctx)
+        if hasattr(template_module, 'sidebar'):
+            doc_ctx['sidebar'] = template_module.sidebar()
+        if hasattr(template_module, 'relbar'):
+            doc_ctx['relbar'] = template_module.relbar()
+
+        if not outfilename:
+            outfilename = path.join(self.outdir, 'pickles',
+                                    os_path(pagename) + self.out_suffix)
+
+        ensuredir(path.dirname(outfilename))
+        f = open(outfilename, 'wb')
+        try:
+            pickle.dump(doc_ctx, f, 2)
+        finally:
+            f.close()
+
+        # if there is a source file, copy the source file for the
+        # "show source" link
+        if ctx.get('sourcename'):
+            source_name = path.join(self.app.builddir, self.app.staticdir,
+                                    '_sources',  os_path(ctx['sourcename']))
+            ensuredir(path.dirname(source_name))
+            copyfile(self.env.doc2path(pagename), source_name)
+
+    def handle_finish(self):
+        StandaloneHTMLBuilder.handle_finish(self)
+        shutil.move(path.join(self.outdir, '_images'),
+                    path.join(self.app.builddir, self.app.staticdir,
+                              '_images'))
+        shutil.move(path.join(self.outdir, '_static'),
+                    path.join(self.app.builddir, self.app.staticdir,
+                              '_static'))
+
+    def dump_search_index(self):
+        self.indexer.finish_indexing()
+
+    def _make_js(self, ctx):
+        def make_script(file):
+            path = ctx['pathto'](file, 1)
+            return '<script type="text/javascript" src="%s"></script>' % path
+
+        opts = """
+<script type="text/javascript">
+  var DOCUMENTATION_OPTIONS = {
+    URL_ROOT: '%s',
+    VERSION: '%s',
+    COLLAPSE_INDEX: false,
+    FILE_SUFFIX: '',
+    HAS_SOURCE: '%s'
+  };
+</script>"""
+        opts = opts % (ctx.get('url_root', ''), escape(ctx['release']),
+                       str(ctx['has_source']).lower())
+        scripts = []
+        for file in ctx['script_files']:
+            scripts.append(make_script(file))
+        scripts.append(make_script('_static/websupport.js'))
+        return opts + '\n' + '\n'.join(scripts)

sphinx/themes/basic/searchresults.html

@@ -0,0 +1,36 @@
+{#
+    basic/searchresults.html
+    ~~~~~~~~~~~~~~~~~
+
+    Template for the body of the search results page.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+#}
+<h1 id="search-documentation">Search</h1>
+<p>
+  From here you can search these documents. Enter your search
+  words into the box below and click "search".
+</p>
+<form action="" method="get">
+  <input type="text" name="q" value="" />
+  <input type="submit" value="search" />
+  <span id="search-progress" style="padding-left: 10px"></span>
+</form>
+{% if search_performed %}
+<h2>Search Results</h2>
+{% if not search_results %}
+<p>'Your search did not match any results.</p>
+{% endif %}
+{% endif %}
+<div id="search-results">
+  {% if search_results %}
+  <ul class="search">
+    {% for href, caption, context in search_results %}
+    <li><a href="{{ href }}?highlight={{ q }}">{{ caption }}</a>
+      <div class="context">{{ context|e }}</div>
+    </li>
+    {% endfor %}
+  </ul>
+  {% endif %}
+</div>

sphinx/themes/basic/static/websupport.js

@@ -0,0 +1,763 @@
+(function($) {
+  $.fn.autogrow = function(){
+    return this.each(function(){
+      var textarea = this;
+
+      $.fn.autogrow.resize(textarea);
+
+      $(textarea)
+	.focus(function() {
+	  textarea.interval = setInterval(function() {
+	    $.fn.autogrow.resize(textarea);
+	  }, 500);
+	})
+	.blur(function() {
+	  clearInterval(textarea.interval);
+	});
+    });
+  };
+
+  $.fn.autogrow.resize = function(textarea) {
+    var lineHeight = parseInt($(textarea).css('line-height'));
+    var lines = textarea.value.split('\n');
+    var columns = textarea.cols;
+    var lineCount = 0;
+    $.each(lines, function() {
+      lineCount += Math.ceil(this.length/columns) || 1;
+    });
+    var height = lineHeight*(lineCount+1);
+    $(textarea).css('height', height);
+  };
+})(jQuery);
+
+(function($) {
+  var commentListEmpty, popup, comp;
+
+  function init() {
+    initTemplates();
+    initEvents();
+    initComparator();
+  };
+
+  function initEvents() {
+    $('a#comment_close').click(function(event) {
+      event.preventDefault();
+      hide();
+    });
+    $('form#comment_form').submit(function(event) {
+      event.preventDefault();
+      addComment($('form#comment_form'));
+    });
+    $('.vote').live("click", function() {
+      handleVote($(this));
+      return false;
+    });
+    $('a.reply').live("click", function() {
+      openReply($(this).attr('id').substring(2));
+      return false;
+    });
+    $('a.close_reply').live("click", function() {
+      closeReply($(this).attr('id').substring(2));
+      return false;
+    });
+    $('a.sort_option').click(function(event) {
+      event.preventDefault();
+      handleReSort($(this));
+    });
+    $('a.show_proposal').live("click", function() {
+      showProposal($(this).attr('id').substring(2));
+      return false;
+    });
+    $('a.hide_proposal').live("click", function() {
+      hideProposal($(this).attr('id').substring(2));
+      return false;
+    });
+    $('a.show_propose_change').live("click", function() {
+      showProposeChange($(this).attr('id').substring(2));
+      return false;
+    });
+    $('a.hide_propose_change').live("click", function() {
+      hideProposeChange($(this).attr('id').substring(2));
+      return false;
+    });
+    $('a.accept_comment').live("click", function() {
+      acceptComment($(this).attr('id').substring(2));
+      return false;
+    });
+    $('a.reject_comment').live("click", function() {
+      rejectComment($(this).attr('id').substring(2));
+      return false;
+    });
+    $('a.delete_comment').live("click", function() {
+      deleteComment($(this).attr('id').substring(2));
+      return false;
+    });
+  };
+
+  function initTemplates() {
+    // Create our popup div, the same div is recycled each time comments
+    // are displayed.
+    popup = $(renderTemplate(popupTemplate, opts));
+    // Setup autogrow on the textareas
+    popup.find('textarea').autogrow();
+    $('body').append(popup);
+  };
+
+  /**
+   * Create a comp function. If the user has preferences stored in
+   * the sortBy cookie, use those, otherwise use the default.
+   */
+  function initComparator() {
+    var by = 'rating'; // Default to sort by rating.
+    // If the sortBy cookie is set, use that instead.
+    if (document.cookie.length > 0) {
+      var start = document.cookie.indexOf('sortBy=');
+      if (start != -1) {
+	start = start + 7;
+	var end = document.cookie.indexOf(";", start);
+	if (end == -1)
+	  end = document.cookie.length;
+	by = unescape(document.cookie.substring(start, end));
+      }
+    }
+    setComparator(by);
+  };
+
+  /**
+   * Show the comments popup window.
+   */
+  function show(nodeId) {
+    var id = nodeId.substring(1);
+
+    // Reset the main comment form, and set the value of the parent input.
+    $('form#comment_form')
+      .find('textarea,input')
+	.removeAttr('disabled').end()
+      .find('input[name="node"]')
+	.val(id).end()
+      .find('textarea[name="proposal"]')
+	.val('')
+	.hide();
+
+    // Position the popup and show it.
+    var clientWidth = document.documentElement.clientWidth;
+    var popupWidth = $('div.popup_comment').width();
+    $('div#focuser').fadeIn('fast');
+    $('div.popup_comment')
+      .css({
+	'top': 100+$(window).scrollTop(),
+	'left': clientWidth/2-popupWidth/2,
+	'position': 'absolute'
+      })
+      .fadeIn('fast', function() {
+	getComments(id);
+      });
+  };
+
+  /**
+   * Hide the comments popup window.
+   */
+  function hide() {
+    $('div#focuser').fadeOut('fast');
+    $('div.popup_comment').fadeOut('fast', function() {
+      $('ul#comment_ul').empty();
+      $('h3#comment_notification').show();
+      $('form#comment_form').find('textarea')
+	.val('').end()
+	.find('textarea, input')
+	  .removeAttr('disabled');
+    });
+  };
+
+  /**
+   * Perform an ajax request to get comments for a node
+   * and insert the comments into the comments tree.
+   */
+  function getComments(id) {
+    $.ajax({
+      type: 'GET',
+      url: opts.getCommentsURL,
+      data: {node: id},
+      success: function(data, textStatus, request) {
+	var ul = $('ul#comment_ul').hide();
+	$('form#comment_form')
+	  .find('textarea[name="proposal"]')
+	    .data('source', data.source);
+
+	if (data.comments.length == 0) {
+	  ul.html('<li>No comments yet.</li>');
+	  commentListEmpty = true;
+	  var speed = 100;
+	}
+	else {
+	  // If there are comments, sort them and put them in the list.
+	  var comments = sortComments(data.comments);
+	  var speed = data.comments.length * 100;
+	  appendComments(comments, ul);
+	  commentListEmpty = false;
+	}
+	$('h3#comment_notification').slideUp(speed+200);
+	ul.slideDown(speed);
+      },
+      error: function(request, textStatus, error) {
+	showError('Oops, there was a problem retrieving the comments.');
+      },
+      dataType: 'json'
+    });
+  };
+
+  /**
+   * Add a comment via ajax and insert the comment into the comment tree.
+   */
+  function addComment(form) {
+    // Disable the form that is being submitted.
+    form.find('textarea,input').attr('disabled', 'disabled');
+    var node_id = form.find('input[name="node"]').val();
+
+    // Send the comment to the server.
+    $.ajax({
+      type: "POST",
+      url: opts.addCommentURL,
+      dataType: 'json',
+      data: {node: node_id,
+             parent: form.find('input[name="parent"]').val(),
+	     text: form.find('textarea[name="comment"]').val(),
+	     proposal: form.find('textarea[name="proposal"]').val()},
+      success: function(data, textStatus, error) {
+	// Reset the form.
+	if (node_id) {
+	  hideProposeChange(node_id);
+	}
+	form.find('textarea')
+	  .val('')
+	    .add(form.find('input'))
+	      .removeAttr('disabled');
+	if (commentListEmpty) {
+	  $('ul#comment_ul').empty();
+	  commentListEmpty = false;
+	}
+	insertComment(data.comment);
+      },
+      error: function(request, textStatus, error) {
+	form.find('textarea,input').removeAttr('disabled');
+	showError('Oops, there was a problem adding the comment.');
+      }
+    });
+  };
+
+  /**
+   * Recursively append comments to the main comment list and children
+   * lists, creating the comment tree.
+   */
+  function appendComments(comments, ul) {
+    $.each(comments, function() {
+      var div = createCommentDiv(this);
+      ul.append($('<li></li>').html(div));
+      appendComments(this.children, div.find('ul.children'));
+      // To avoid stagnating data, don't store the comments children in data.
+      this.children = null;
+      div.data('comment', this);
+    });
+  };
+
+  /**
+   * After adding a new comment, it must be inserted in the correct
+   * location in the comment tree.
+   */
+  function insertComment(comment) {
+    var div = createCommentDiv(comment);
+
+    // To avoid stagnating data, don't store the comments children in data.
+    comment.children = null;
+    div.data('comment', comment);
+
+    if (comment.node != null) {
+      var ul = $('ul#comment_ul');
+      var siblings = getChildren(ul);
+    }
+    else {
+      var ul = $('#cl' + comment.parent);
+      var siblings = getChildren(ul);
+    }
+
+    var li = $('<li></li>');
+    li.hide();
+
+    // Determine where in the parents children list to insert this comment.
+    for(i=0; i < siblings.length; i++) {
+      if (comp(comment, siblings[i]) <= 0) {
+	$('#cd' + siblings[i].id)
+	  .parent()
+	    .before(li.html(div));
+	li.slideDown('fast');
+	return;
+      }
+    }
+
+    // If we get here, this comment rates lower than all the others,
+    // or it is the only comment in the list.
+    ul.append(li.html(div));
+    li.slideDown('fast');
+  };
+
+  function acceptComment(id) {
+    $.ajax({
+      type: 'POST',
+      url: opts.acceptCommentURL,
+      data: {id: id},
+      success: function(data, textStatus, request) {
+	$('#cm' + id).fadeOut('fast');
+      },
+      error: function(request, textStatus, error) {
+	showError("Oops, there was a problem accepting the comment.");
+      },
+    });
+  };
+
+  function rejectComment(id) {
+    $.ajax({
+      type: 'POST',
+      url: opts.rejectCommentURL,
+      data: {id: id},
+      success: function(data, textStatus, request) {
+	var div = $('#cd' + id);
+	div.slideUp('fast', function() {
+	  div.remove();
+	});
+      },
+      error: function(request, textStatus, error) {
+	showError("Oops, there was a problem rejecting the comment.");
+      },
+    });
+  };
+
+  function deleteComment(id) {
+    $.ajax({
+      type: 'POST',
+      url: opts.deleteCommentURL,
+      data: {id: id},
+      success: function(data, textStatus, request) {
+	var div = $('#cd' + id);
+	div
+	  .find('span.user_id:first')
+	    .text('[deleted]').end()
+	  .find('p.comment_text:first')
+	    .text('[deleted]').end()
+	  .find('#cm' + id + ', #dc' + id + ', #ac' + id + ', #rc' + id +
+		', #sp' + id + ', #hp' + id + ', #cr' + id + ', #rl' + id)
+	    .remove();
+	var comment = div.data('comment');
+	comment.username = '[deleted]';
+	comment.text = '[deleted]';
+	div.data('comment', comment);
+      },
+      error: function(request, textStatus, error) {
+	showError("Oops, there was a problem deleting the comment.");
+      },
+    });
+  };
+
+  function showProposal(id) {
+    $('#sp' + id).hide();
+    $('#hp' + id).show();
+    $('#pr' + id).slideDown('fast');
+  };
+
+  function hideProposal(id) {
+    $('#hp' + id).hide();
+    $('#sp' + id).show();
+    $('#pr' + id).slideUp('fast');
+  };
+
+  function showProposeChange(id) {
+    $('a.show_propose_change').hide();
+    $('a.hide_propose_change').show();
+    var textarea = $('textarea[name="proposal"]');
+    textarea.val(textarea.data('source'));
+    $.fn.autogrow.resize(textarea[0]);
+    textarea.slideDown('fast');
+  };
+
+  function hideProposeChange(id) {
+    $('a.hide_propose_change').hide();
+    $('a.show_propose_change').show();
+    var textarea = $('textarea[name="proposal"]');
+    textarea.val('').removeAttr('disabled');
+    textarea.slideUp('fast');
+  };
+
+  /**
+   * Handle when the user clicks on a sort by link.
+   */
+  function handleReSort(link) {
+    setComparator(link.attr('id'));
+     // Save/update the sortBy cookie.
+    var expiration = new Date();
+    expiration.setDate(expiration.getDate() + 365);
+    document.cookie= 'sortBy=' + escape(link.attr('id')) +
+      ';expires=' + expiration.toUTCString();
+    var comments = getChildren($('ul#comment_ul'), true);
+    comments = sortComments(comments);
+
+    appendComments(comments, $('ul#comment_ul').empty());
+  };
+
+  /**
+   * Function to process a vote when a user clicks an arrow.
+   */
+  function handleVote(link) {
+    if (!opts.voting) {
+      showError("You'll need to login to vote.");
+      return;
+    }
+
+    var id = link.attr('id');
+    // If it is an unvote, the new vote value is 0,
+    // Otherwise it's 1 for an upvote, or -1 for a downvote.
+    if (id.charAt(1) == 'u')
+      var value = 0;
+    else
+      var value = id.charAt(0) == 'u' ? 1 : -1;
+
+    // The data to be sent to the server.
+    var d = {
+      comment_id: id.substring(2),
+      value: value
+    };
+
+    // Swap the vote and unvote links.
+    link.hide();
+    $('#' + id.charAt(0) + (id.charAt(1) == 'u' ? 'v' : 'u') + d.comment_id)
+      .show();
+
+    // The div the comment is displayed in.
+    var div = $('div#cd' + d.comment_id);
+    var data = div.data('comment');
+
+    // If this is not an unvote, and the other vote arrow has
+    // already been pressed, unpress it.
+    if ((d.value != 0) && (data.vote == d.value*-1)) {
+      $('#' + (d.value == 1 ? 'd' : 'u') + 'u' + d.comment_id)
+	.hide();
+      $('#' + (d.value == 1 ? 'd' : 'u') + 'v' + d.comment_id)
+	.show();
+    }
+
+    // Update the comments rating in the local data.
+    data.rating += (data.vote == 0) ? d.value : (d.value - data.vote);
+    data.vote = d.value;
+    div.data('comment', data);
+
+    // Change the rating text.
+    div.find('.rating:first')
+      .text(data.rating + ' point' + (data.rating == 1 ? '' : 's'));
+
+    // Send the vote information to the server.
+    $.ajax({
+      type: "POST",
+      url: opts.processVoteURL,
+      data: d,
+      error: function(request, textStatus, error) {
+	showError("Oops, there was a problem casting that vote.");
+      }
+    });
+  };
+
+  /**
+   * Open a reply form used to reply to an existing comment.
+   */
+  function openReply(id) {
+    // Swap out the reply link for the hide link
+    $('#rl' + id).hide();
+    $('#cr' + id).show();
+
+    // Add the reply li to the children ul.
+    var div = $(renderTemplate(replyTemplate, {id: id})).hide();
+    $('#cl' + id)
+      .prepend(div)
+      // Setup the submit handler for the reply form.
+      .find('#rf' + id)
+	.submit(function(event) {
+	  event.preventDefault();
+	  addComment($('#rf' + id));
+	  closeReply(id);
+	});
+    div.slideDown('fast');
+  };
+
+  /**
+   * Close the reply form opened with openReply.
+   */
+  function closeReply(id) {
+    // Remove the reply div from the DOM.
+    $('#rd' + id).slideUp('fast', function() {
+      $(this).remove();
+    });
+
+    // Swap out the hide link for the reply link
+    $('#cr' + id).hide();
+    $('#rl' + id).show();
+  };
+
+  /**
+   * Recursively sort a tree of comments using the comp comparator.
+   */
+  function sortComments(comments) {
+    comments.sort(comp);
+    $.each(comments, function() {
+      this.children = sortComments(this.children);
+    });
+    return comments;
+  };
+
+  /**
+   * Set comp, which is a comparator function used for sorting and
+   * inserting comments into the list.
+   */
+  function setComparator(by) {
+    // If the first three letters are "asc", sort in ascending order
+    // and remove the prefix.
+    if (by.substring(0,3) == 'asc') {
+      var i = by.substring(3);
+      comp = function(a, b) { return a[i] - b[i]; }
+    }
+    // Otherwise sort in descending order.
+    else
+      comp = function(a, b) { return b[by] - a[by]; }
+
+    // Reset link styles and format the selected sort option.
+    $('a.sel').attr('href', '#').removeClass('sel');
+    $('#' + by).removeAttr('href').addClass('sel');
+  };
+
+  /**
+   * Get the children comments from a ul. If recursive is true,
+   * recursively include childrens' children.
+   */
+  function getChildren(ul, recursive) {
+    var children = [];
+    ul.children().children("[id^='cd']")
+      .each(function() {
+	var comment = $(this).data('comment');
+	if (recursive) {
+	  comment.children =
+	    getChildren($(this).find('#cl' + comment.id), true);
+	}
+	children.push(comment);
+      });
+    return children;
+  };
+
+  /**
+   * Create a div to display a comment in.
+   */
+  function createCommentDiv(comment) {
+    // Prettify the comment rating.
+    comment.pretty_rating = comment.rating + ' point' +
+      (comment.rating == 1 ? '' : 's');
+    // Create a div for this comment.
+    var context = $.extend({}, opts, comment);
+    var div = $(renderTemplate(commentTemplate, context));
+
+    // If the user has voted on this comment, highlight the correct arrow.
+    if (comment.vote) {
+      var direction = (comment.vote == 1) ? 'u' : 'd';
+      div.find('#' + direction + 'v' + comment.id).hide();
+      div.find('#' + direction + 'u' + comment.id).show();
+    }
+
+    if (comment.text != '[deleted]') {
+      div.find('a.reply').show();
+      if (comment.proposal_diff) {
+	div.find('#sp' + comment.id).show();
+      }
+      if (opts.moderator && !comment.displayed) {
+	div.find('#cm' + comment.id).show();
+      }
+      if (opts.moderator || (opts.username == comment.username)) {
+	div.find('#dc' + comment.id).show();
+      }
+    }
+
+    return div;
+  }
+
+  /**
+   * A simple template renderer. Placeholders such as <%id%> are replaced
+   * by context['id']. Items are always escaped.
+   */
+  function renderTemplate(template, context) {
+    var esc = $('<span></span>');
+
+    function handle(ph, escape) {
+      var cur = context;
+      $.each(ph.split('.'), function() {
+	cur = cur[this];
+      });
+      return escape ? esc.text(cur || "").html() : cur;
+    }
+
+    return template.replace(/<([%#])([\w\.]*)\1>/g, function(){
+      return handle(arguments[2], arguments[1] == '%' ? true : false);
+    });
+  };
+
+  function showError(message) {
+    $('<div class="popup_error">' +
+      '<h1>' + message + '</h1>' +
+      '</div>')
+	.appendTo('body')
+	  .fadeIn("slow")
+	    .delay(2000)
+	      .fadeOut("slow");
+  };
+
+  /**
+   * Add a link the user uses to open the comments popup.
+   */
+  $.fn.comment = function() {
+    return this.each(function() {
+      var id = $(this).attr('id').substring(1);
+      var count = COMMENT_METADATA[id]
+      var title = count + ' comment' + (count == 1 ? '' : 's');
+      var image = count > 0 ? opts.commentBrightImage : opts.commentImage;
+      $(this).append(
+	$('<a href="#" class="sphinx_comment"></a>')
+	  .html('<img src="' + image + '" alt="comment" />')
+	  .attr('title', title)
+	  .click(function(event) {
+	    event.preventDefault();
+	    show($(this).parent().attr('id'));
+	  }));
+    });
+  };
+
+  var replyTemplate = '    <li>\
+      <div class="reply_div" id="rd<%id%>">\
+	<form id="rf<%id%>">\
+	  <textarea name="comment" cols="80"></textarea>\
+          <input type="submit" value="add reply" />\
+          <input type="hidden" name="parent" value="<%id%>" />\
+          <input type="hidden" name="node" value="" />\
+        </form>\
+      </div>\
+    </li>';
+
+  var commentTemplate = '    <div  id="cd<%id%>" class="spxcdiv">\
+      <div class="vote">\
+	<div class="arrow">\
+	  <a href="#" id="uv<%id%>" class="vote">\
+	    <img src="<%upArrow%>" />\
+	  </a>\
+	  <a href="#" id="uu<%id%>" class="un vote">\
+	    <img src="<%upArrowPressed%>" />\
+	  </a>\
+	</div>\
+      <div class="arrow">\
+	  <a href="#" id="dv<%id%>" class="vote">\
+	    <img src="<%downArrow%>" id="da<%id%>" />\
+	  </a>\
+	  <a href="#" id="du<%id%>" class="un vote">\
+	    <img src="<%downArrowPressed%>" />\
+	  </a>\
+	</div>\
+      </div>\
+      <div class="comment_content">\
+	<p class="tagline comment">\
+	  <span class="user_id"><%username%></span>\
+	  <span class="rating"><%pretty_rating%></span>\
+	  <span class="delta"><%time.delta%></span>\
+	</p>\
+	<p class="comment_text comment"><%text%></p>\
+	<p class="comment_opts comment">\
+	  <a href="#" class="reply hidden" id="rl<%id%>">reply &#9657;</a>\
+	  <a href="#" class="close_reply" id="cr<%id%>">reply &#9663;</a>\
+	  <a href="#" id="sp<%id%>" class="show_proposal">\
+	    proposal &#9657;\
+	  </a>\
+	  <a href="#" id="hp<%id%>" class="hide_proposal">\
+	    proposal &#9663;\
+	  </a>\
+	  <a href="#" id="dc<%id%>" class="delete_comment hidden">\
+	    delete\
+	  </a>\
+	  <span id="cm<%id%>" class="moderation hidden">\
+	    <a href="#" id="ac<%id%>" class="accept_comment">accept</a>\
+	    <a href="#" id="rc<%id%>" class="reject_comment">reject</a>\
+	  </span>\
+	</p>\
+	<pre class="proposal" id="pr<%id%>">\
+<#proposal_diff#>\
+	</pre>\
+	<ul class="children" id="cl<%id%>"></ul>\
+      </div>\
+      <div class="clearleft"></div>\
+    </div>';
+
+   var popupTemplate = '    <div class="popup_comment">\
+      <a id="comment_close" href="#">x</a>\
+      <h1>Comments</h1>\
+      <form method="post" id="comment_form" action="/docs/add_comment">\
+	<textarea name="comment" cols="80"></textarea>\
+	<p class="propose_button">\
+	  <a href="#" class="show_propose_change">\
+	    Propose a change &#9657;\
+	  </a>\
+	  <a href="#" class="hide_propose_change">\
+	    Propose a change &#9663;\
+	  </a>\
+	</p>\
+	<textarea name="proposal" cols="80" spellcheck="false"></textarea>\
+	<input type="submit" value="add comment" id="comment_button" />\
+	<input type="hidden" name="node" />\
+	<input type="hidden" name="parent" value="" />\
+      <p class="sort_options">\
+	Sort by:\
+	<a href="#" class="sort_option" id="rating">top</a>\
+	<a href="#" class="sort_option" id="ascage">newest</a>\
+	<a href="#" class="sort_option" id="age">oldest</a>\
+      </p>\
+      </form>\
+      <h3 id="comment_notification">loading comments... <img src="' +
+      opts.loadingImage + '" alt="" /></h3>\
+      <ul id="comment_ul"></ul>\
+    </div>\
+    <div id="focuser"></div>';
+
+
+  var opts = jQuery.extend({
+    processVoteURL: '/process_vote',
+    addCommentURL: '/add_comment',
+    getCommentsURL: '/get_comments',
+    acceptCommentURL: '/accept_comment',
+    rejectCommentURL: '/reject_comment',
+    rejectCommentURL: '/delete_comment',
+    commentImage: '/static/_static/comment.png',
+    loadingImage: '/static/_static/comment.png',
+    commentBrightImage: '/static/_static/comment-bright.png',
+    upArrow: '/static/_static/up.png',
+    downArrow: '/static/_static/down.png',
+    upArrowPressed: '/static/_static/up-pressed.png',
+    downArrowPressed: '/static/_static/down-pressed.png',
+    voting: false,
+    moderator: false
+  }, COMMENT_OPTIONS);
+
+  $(document).ready(function() {
+    init();
+  });
+})(jQuery);
+
+$(document).ready(function() {
+  $('.spxcmt').comment();
+
+  /** Highlight search words in search results. */
+  $("div.context").each(function() {
+    var params = $.getQueryParameters();
+    var terms = (params.q) ? params.q[0].split(/\s+/) : [];
+    var result = $(this);
+    $.each(terms, function() {
+      result.highlightText(this.toLowerCase(), 'highlighted');
+    });
+  });
+});

sphinx/websupport/__init__.py

@@ -0,0 +1,407 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.websupport
+    ~~~~~~~~~~~~~~~~~
+
+    Base Module for web support functions.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+import sys
+import cPickle as pickle
+import posixpath
+from os import path
+from datetime import datetime
+
+from jinja2 import Environment, FileSystemLoader
+
+from sphinx.application import Sphinx
+from sphinx.util.osutil import ensuredir
+from sphinx.util.jsonimpl import dumps as dump_json
+from sphinx.websupport.search import BaseSearch, search_adapters
+from sphinx.websupport.storage import StorageBackend
+from sphinx.websupport.errors import *
+
+class WebSupportApp(Sphinx):
+    def __init__(self, *args, **kwargs):
+        self.staticdir = kwargs.pop('staticdir', None)
+        self.builddir = kwargs.pop('builddir', None)
+        self.search = kwargs.pop('search', None)
+        self.storage = kwargs.pop('storage', None)
+        Sphinx.__init__(self, *args, **kwargs)
+
+class WebSupport(object):
+    """The main API class for the web support package. All interactions
+    with the web support package should occur through this class.
+    """
+    def __init__(self, srcdir='', builddir='', datadir='', search=None,
+                 storage=None, status=sys.stdout, warning=sys.stderr,
+                 moderation_callback=None, staticdir='static',
+                 docroot=''):
+        self.srcdir = srcdir
+        self.builddir = builddir
+        self.outdir = path.join(builddir, 'data')
+        self.datadir = datadir or self.outdir
+        self.staticdir = staticdir.strip('/')
+        self.docroot = docroot.strip('/')
+        self.status = status
+        self.warning = warning
+        self.moderation_callback = moderation_callback
+
+        self._init_templating()
+        self._init_search(search)
+        self._init_storage(storage)
+
+        self._make_base_comment_options()
+
+    def _init_storage(self, storage):
+        if isinstance(storage, StorageBackend):
+            self.storage = storage
+        else:
+            # If a StorageBackend isn't provided, use the default
+            # SQLAlchemy backend.
+            from sphinx.websupport.storage.sqlalchemystorage \
+                import SQLAlchemyStorage
+            from sqlalchemy import create_engine
+            db_path = path.join(self.datadir, 'db', 'websupport.db')
+            ensuredir(path.dirname(db_path))
+            uri = storage or 'sqlite:///%s' % db_path
+            engine = create_engine(uri)
+            self.storage = SQLAlchemyStorage(engine)
+
+    def _init_templating(self):
+        import sphinx
+        template_path = path.join(path.dirname(sphinx.__file__),
+                                  'themes', 'basic')
+        loader = FileSystemLoader(template_path)
+        self.template_env = Environment(loader=loader)
+
+    def _init_search(self, search):
+        if isinstance(search, BaseSearch):
+            self.search = search
+        else:
+            mod, cls = search_adapters[search or 'null']
+            mod = 'sphinx.websupport.search.' + mod
+            SearchClass = getattr(__import__(mod, None, None, [cls]), cls)
+            search_path = path.join(self.datadir, 'search')
+            self.search = SearchClass(search_path)
+        self.results_template = \
+            self.template_env.get_template('searchresults.html')
+
+    def build(self):
+        """Build the documentation. Places the data into the `outdir`
+        directory. Use it like this::
+
+            support = WebSupport(srcdir, builddir, search='xapian')
+            support.build()
+
+        This will read reStructured text files from `srcdir`. Then it will
+        build the pickles and search index, placing them into `builddir`.
+        It will also save node data to the database.
+        """
+        if not self.srcdir:
+            raise SrcdirNotSpecifiedError( \
+                'No srcdir associated with WebSupport object')
+        doctreedir = path.join(self.outdir, 'doctrees')
+        app = WebSupportApp(self.srcdir, self.srcdir,
+                            self.outdir, doctreedir, 'websupport',
+                            search=self.search, status=self.status,
+                            warning=self.warning, storage=self.storage,
+                            staticdir=self.staticdir, builddir=self.builddir)
+
+        self.storage.pre_build()
+        app.build()
+        self.storage.post_build()
+
+    def get_document(self, docname, username='', moderator=False):
+        """Load and return a document from a pickle. The document will
+        be a dict object which can be used to render a template::
+
+            support = WebSupport(datadir=datadir)
+            support.get_document('index', username, moderator)
+
+        In most cases `docname` will be taken from the request path and
+        passed directly to this function. In Flask, that would be something
+        like this::
+
+            @app.route('/<path:docname>')
+            def index(docname):
+                username = g.user.name if g.user else ''
+                moderator = g.user.moderator if g.user else False
+                try:
+                    document = support.get_document(docname, username,
+                                                    moderator)
+                except DocumentNotFoundError:
+                    abort(404)
+                render_template('doc.html', document=document)
+
+        The document dict that is returned contains the following items
+        to be used during template rendering.
+
+        * **body**: The main body of the document as HTML
+        * **sidebar**: The sidebar of the document as HTML
+        * **relbar**: A div containing links to related documents
+        * **title**: The title of the document
+        * **css**: Links to css files used by Sphinx
+        * **js**: Javascript containing comment options
+
+        This raises :class:`~sphinx.websupport.errors.DocumentNotFoundError`
+        if a document matching `docname` is not found.
+
+        :param docname: the name of the document to load.
+        """
+        infilename = path.join(self.datadir, 'pickles', docname + '.fpickle')
+
+        try:
+            f = open(infilename, 'rb')
+        except IOError:
+            raise DocumentNotFoundError(
+                'The document "%s" could not be found' % docname)
+
+        document = pickle.load(f)
+        comment_opts = self._make_comment_options(username, moderator)
+        comment_metadata = self.storage.get_metadata(docname, moderator)
+
+        document['js'] = '\n'.join([comment_opts,
+                                    self._make_metadata(comment_metadata),
+                                    document['js']])
+        return document
+
+    def get_search_results(self, q):
+        """Perform a search for the query `q`, and create a set
+        of search results. Then render the search results as html and
+        return a context dict like the one created by
+        :meth:`get_document`::
+
+            document = support.get_search_results(q)
+
+        :param q: the search query
+        """
+        results = self.search.query(q)
+        ctx = {'search_performed': True,
+               'search_results': results,
+               'q': q}
+        document = self.get_document('search')
+        document['body'] = self.results_template.render(ctx)
+        document['title'] = 'Search Results'
+        return document
+
+    def get_data(self, node_id, username=None, moderator=False):
+        """Get the comments and source associated with `node_id`. If
+        `username` is given vote information will be included with the
+        returned comments. The default CommentBackend returns a dict with
+        two keys, *source*, and *comments*. *source* is raw source of the
+        node and is used as the starting point for proposals a user can
+        add. *comments* is a list of dicts that represent a comment, each
+        having the following items:
+
+        ============= ======================================================
+        Key           Contents
+        ============= ======================================================
+        text          The comment text.
+        username      The username that was stored with the comment.
+        id            The comment's unique identifier.
+        rating        The comment's current rating.
+        age           The time in seconds since the comment was added.
+        time          A dict containing time information. It contains the
+                      following keys: year, month, day, hour, minute, second,
+                      iso, and delta. `iso` is the time formatted in ISO
+                      8601 format. `delta` is a printable form of how old
+                      the comment is (e.g. "3 hours ago").
+        vote          If `user_id` was given, this will be an integer
+                      representing the vote. 1 for an upvote, -1 for a
+                      downvote, or 0 if unvoted.
+        node          The id of the node that the comment is attached to.
+                      If the comment's parent is another comment rather than
+                      a node, this will be null.
+        parent        The id of the comment that this comment is attached
+                      to if it is not attached to a node.
+        children      A list of all children, in this format.
+        proposal_diff An HTML representation of the differences between the
+                      the current source and the user's proposed source.
+        ============= ======================================================
+
+        :param node_id: the id of the node to get comments for.
+        :param username: the username of the user viewing the comments.
+        :param moderator: whether the user is a moderator.
+        """
+        return self.storage.get_data(node_id, username, moderator)
+
+    def delete_comment(self, comment_id, username='', moderator=False):
+        """Delete a comment. Doesn't actually delete the comment, but
+        instead replaces the username and text files with "[deleted]" so
+        as not to leave any comments orphaned.
+
+        If `moderator` is True, the comment will always be deleted. If
+        `moderator` is False, the comment will only be deleted if the
+        `username` matches the `username` on the comment.
+
+        This raises :class:`~sphinx.websupport.errors.UserNotAuthorizedError`
+        if moderator is False and `username` doesn't match username on the
+        comment.
+
+        :param comment_id: the id of the comment to delete.
+        :param username: the username requesting the deletion.
+        :param moderator: whether the requestor is a moderator.
+        """
+        self.storage.delete_comment(comment_id, username, moderator)
+
+    def add_comment(self, text, node_id='', parent_id='', displayed=True,
+                    username=None, time=None, proposal=None,
+                    moderator=False):
+        """Add a comment to a node or another comment. Returns the comment
+        in the same format as :meth:`get_comments`. If the comment is being
+        attached to a node, pass in the node's id (as a string) with the
+        node keyword argument::
+
+            comment = support.add_comment(text, node_id=node_id)
+
+        If the comment is the child of another comment, provide the parent's
+        id (as a string) with the parent keyword argument::
+
+            comment = support.add_comment(text, parent_id=parent_id)
+
+        If you would like to store a username with the comment, pass
+        in the optional `username` keyword argument::
+
+            comment = support.add_comment(text, node=node_id,
+                                          username=username)
+
+        :param parent_id: the prefixed id of the comment's parent.
+        :param text: the text of the comment.
+        :param displayed: for moderation purposes
+        :param username: the username of the user making the comment.
+        :param time: the time the comment was created, defaults to now.
+        """
+        comment = self.storage.add_comment(text, displayed, username,
+                                           time, proposal, node_id,
+                                           parent_id, moderator)
+        if not displayed and self.moderation_callback:
+            self.moderation_callback(comment)
+        return comment
+
+    def process_vote(self, comment_id, username, value):
+        """Process a user's vote. The web support package relies
+        on the API user to perform authentication. The API user will
+        typically receive a comment_id and value from a form, and then
+        make sure the user is authenticated. A unique username  must be
+        passed in, which will also be used to retrieve the user's past
+        voting data. An example, once again in Flask::
+
+            @app.route('/docs/process_vote', methods=['POST'])
+            def process_vote():
+                if g.user is None:
+                    abort(401)
+                comment_id = request.form.get('comment_id')
+                value = request.form.get('value')
+                if value is None or comment_id is None:
+                    abort(400)
+                support.process_vote(comment_id, g.user.name, value)
+                return "success"
+
+        :param comment_id: the comment being voted on
+        :param username: the unique username of the user voting
+        :param value: 1 for an upvote, -1 for a downvote, 0 for an unvote.
+        """
+        value = int(value)
+        if not -1 <= value <= 1:
+            raise ValueError('vote value %s out of range (-1, 1)' % value)
+        self.storage.process_vote(comment_id, username, value)
+
+    def update_username(self, old_username, new_username):
+        """To remain decoupled from a webapp's authentication system, the
+        web support package stores a user's username with each of their
+        comments and votes. If the authentication system allows a user to
+        change their username, this can lead to stagnate data in the web
+        support system. To avoid this, each time a username is changed, this
+        method should be called.
+
+        :param old_username: The original username.
+        :param new_username: The new username.
+        """
+        self.storage.update_username(old_username, new_username)
+
+    def accept_comment(self, comment_id, moderator=False):
+        """Accept a comment that is pending moderation.
+
+        This raises :class:`~sphinx.websupport.errors.UserNotAuthorizedError`
+        if moderator is False.
+
+        :param comment_id: The id of the comment that was accepted.
+        :param moderator: Whether the user making the request is a moderator.
+        """
+        if not moderator:
+            raise UserNotAuthorizedError()
+        self.storage.accept_comment(comment_id)
+
+    def reject_comment(self, comment_id, moderator=False):
+        """Reject a comment that is pending moderation.
+
+        This raises :class:`~sphinx.websupport.errors.UserNotAuthorizedError`
+        if moderator is False.
+
+        :param comment_id: The id of the comment that was accepted.
+        :param moderator: Whether the user making the request is a moderator.
+        """
+        if not moderator:
+            raise UserNotAuthorizedError()
+        self.storage.reject_comment(comment_id)
+
+    def _make_base_comment_options(self):
+        """Helper method to create the part of the COMMENT_OPTIONS javascript
+        that remains the same throughout the lifetime of the
+        :class:`~sphinx.websupport.WebSupport` object.
+        """
+        comment_urls = [
+            ('addCommentURL', 'add_comment'),
+            ('getCommentsURL', 'get_comments'),
+            ('processVoteURL', 'process_vote'),
+            ('acceptCommentURL', 'accept_comment'),
+            ('rejectCommentURL', 'reject_comment'),
+            ('deleteCommentURL', 'delete_comment')
+        ]
+        static_urls = [
+            ('commentImage', 'comment.png'),
+            ('loadingImage', 'ajax-loader.gif'),
+            ('commentBrightImage', 'comment-bright.png'),
+            ('upArrow', 'up.png'),
+            ('upArrowPressed', 'up-pressed.png'),
+            ('downArrow', 'down.png'),
+            ('downArrowPressed', 'down-pressed.png')
+        ]
+
+        self.base_comment_opts = {}
+        for key, value in comment_urls:
+            self.base_comment_opts[key] = posixpath.join(self.docroot, value)
+        for key, value in static_urls:
+            self.base_comment_opts[key] = posixpath.join(self.staticdir, value)
+
+    def _make_comment_options(self, username, moderator):
+        """Helper method to create the parts of the COMMENT_OPTIONS
+        javascript that are unique to each request.
+
+        :param username: The username of the user making the request.
+        :param moderator: Whether the user making the request is a moderator.
+        """
+        parts = [self.base_comment_opts]
+        rv = self.base_comment_opts.copy()
+        if username:
+            rv.update({
+                'voting': True,
+                'username': username,
+                'moderator': str(moderator).lower(),
+            })
+        return '\n'.join([
+            '<script type="text/javascript">',
+            'var COMMENT_OPTIONS = %s;' % dump_json(rv),
+            '</script>'
+        ])
+
+    def _make_metadata(self, data):
+        return '\n'.join([
+            '<script type="text/javascript">',
+            'var COMMENT_METADATA = %s;' % dump_json(data),
+            '</script>'
+        ])

sphinx/websupport/errors.py

@@ -0,0 +1,33 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.websupport.errors
+    ~~~~~~~~~~~~~~~~~~~~~~~~
+
+    Contains Error classes for the web support package.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+__all__ = ['DocumentNotFoundError', 'SrcdirNotSpecifiedError',
+           'UserNotAuthorizedError', 'CommentNotAllowedError',
+           'NullSearchException']
+
+class DocumentNotFoundError(Exception):
+    pass
+
+
+class SrcdirNotSpecifiedError(Exception):
+    pass
+
+
+class UserNotAuthorizedError(Exception):
+    pass
+
+
+class CommentNotAllowedError(Exception):
+    pass
+
+
+class NullSearchException(Exception):
+    pass

sphinx/websupport/search/__init__.py

@@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.websupport.search
+    ~~~~~~~~~~~~~~~~~~~~~~~~
+
+    Server side search support for the web support package.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+import re
+
+class BaseSearch(object):
+    def __init__(self, path):
+        pass
+
+    def init_indexing(self, changed=[]):
+        """Called by the builder to initialize the search indexer. `changed`
+        is a list of pagenames that will be reindexed. You may want to remove
+        these from the search index before indexing begins.
+
+        :param changed: a list of pagenames that will be re-indexed
+        """
+        pass
+
+    def finish_indexing(self):
+        """Called by the builder when writing has been completed. Use this
+        to perform any finalization or cleanup actions after indexing is
+        complete.
+        """
+        pass
+
+    def feed(self, pagename, title, doctree):
+        """Called by the builder to add a doctree to the index. Converts the
+        `doctree` to text and passes it to :meth:`add_document`. You probably
+        won't want to override this unless you need access to the `doctree`.
+        Override :meth:`add_document` instead.
+
+        :param pagename: the name of the page to be indexed
+        :param title: the title of the page to be indexed
+        :param doctree: is the docutils doctree representation of the page
+        """
+        self.add_document(pagename, title, doctree.astext())
+
+    def add_document(self, pagename, title, text):
+        """Called by :meth:`feed` to add a document to the search index.
+        This method should should do everything necessary to add a single
+        document to the search index.
+
+        `pagename` is name of the page being indexed. It is the combination
+        of the source files relative path and filename,
+        minus the extension. For example, if the source file is
+        "ext/builders.rst", the `pagename` would be "ext/builders". This
+        will need to be returned with search results when processing a
+        query.
+
+        :param pagename: the name of the page being indexed
+        :param title: the page's title
+        :param text: the full text of the page
+        """
+        raise NotImplementedError()
+
+    def query(self, q):
+        """Called by the web support api to get search results. This method
+        compiles the regular expression to be used when
+        :meth:`extracting context <extract_context>`, then calls
+        :meth:`handle_query`. You won't want to override this unless you
+        don't want to use the included :meth:`extract_context` method.
+        Override :meth:`handle_query` instead.
+
+        :param q: the search query string.
+        """
+        self.context_re = re.compile('|'.join(q.split()), re.I)
+        return self.handle_query(q)
+
+    def handle_query(self, q):
+        """Called by :meth:`query` to retrieve search results for a search
+        query `q`. This should return an iterable containing tuples of the
+        following format::
+
+            (<path>, <title>, <context>)
+
+        `path` and `title` are the same values that were passed to
+        :meth:`add_document`, and `context` should be a short text snippet
+        of the text surrounding the search query in the document.
+
+        The :meth:`extract_context` method is provided as a simple way
+        to create the `context`.
+
+        :param q: the search query
+        """
+        raise NotImplementedError()
+
+    def extract_context(self, text, length=240):
+        """Extract the context for the search query from the documents
+        full `text`.
+
+        :param text: the full text of the document to create the context for
+        :param length: the length of the context snippet to return.
+        """
+        res = self.context_re.search(text)
+        if res is None:
+            return ''
+        context_start = max(res.start() - length/2, 0)
+        context_end = context_start + length
+        context = ''.join(['...' if context_start > 0 else '',
+                           text[context_start:context_end],
+                           '...' if context_end < len(text) else ''])
+
+        try:
+            return unicode(context, errors='ignore')
+        except TypeError:
+            return context
+
+# The build in search adapters.
+search_adapters = {
+    'xapian': ('xapiansearch', 'XapianSearch'),
+    'whoosh': ('whooshsearch', 'WhooshSearch'),
+    'null': ('nullsearch', 'NullSearch')
+    }

sphinx/websupport/search/nullsearch.py

@@ -0,0 +1,23 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.websupport.search.nullsearch
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    The default search adapter, does nothing.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+from sphinx.websupport.search import BaseSearch
+from sphinx.websupport.errors import *
+
+class NullSearch(BaseSearch):
+    """A search adapter that does nothing. Used when no search adapter
+    is specified.
+    """
+    def feed(self, pagename, title, doctree):
+        pass
+
+    def query(self, q):
+        raise NullSearchException('No search adapter specified.')

sphinx/websupport/search/whooshsearch.py

@@ -0,0 +1,57 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.websupport.search.whooshsearch
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    Whoosh search adapter.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+from whoosh import index
+from whoosh.fields import Schema, ID, TEXT, STORED
+from whoosh.analysis import StemmingAnalyzer
+from whoosh import highlight
+
+from sphinx.util.osutil import ensuredir
+from sphinx.websupport.search import BaseSearch
+
+class WhooshSearch(BaseSearch):
+    """The whoosh search adapter for sphinx web support."""
+
+    # Define the Whoosh Schema for the search index.
+    schema = Schema(path=ID(stored=True, unique=True),
+                    title=TEXT(field_boost=2.0, stored=True),
+                    text=TEXT(analyzer=StemmingAnalyzer(), stored=True))
+
+    def __init__(self, db_path):
+        ensuredir(db_path)
+        if index.exists_in(db_path):
+            self.index = index.open_dir(db_path)
+        else:
+            self.index = index.create_in(db_path, schema=self.schema)
+
+    def init_indexing(self, changed=[]):
+        for changed_path in changed:
+            self.index.delete_by_term('path', changed_path)
+        self.index_writer = self.index.writer()
+
+    def finish_indexing(self):
+        self.index_writer.commit()
+
+    def add_document(self, pagename, title, text):
+        self.index_writer.add_document(path=unicode(pagename),
+                                       title=title,
+                                       text=text)
+
+    def handle_query(self, q):
+        searcher = self.index.searcher()
+        whoosh_results = searcher.find('text', q)
+        results = []
+        for result in whoosh_results:
+            context = self.extract_context(result['text'])
+            results.append((result['path'],
+                            result.get('title', ''),
+                            context))
+        return results

sphinx/websupport/search/xapiansearch.py

@@ -0,0 +1,82 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.websupport.search.xapiansearch
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    Xapian search adapter.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+from os import path
+
+import xapian
+
+from sphinx.util.osutil import ensuredir
+from sphinx.websupport.search import BaseSearch
+
+class XapianSearch(BaseSearch):
+    # Adapted from the GSOC 2009 webapp project.
+
+    # Xapian metadata constants
+    DOC_PATH = 0
+    DOC_TITLE = 1
+
+    def __init__(self, db_path):
+        self.db_path = db_path
+
+    def init_indexing(self, changed=[]):
+        ensuredir(self.db_path)
+        self.database = xapian.WritableDatabase(self.db_path,
+                                                xapian.DB_CREATE_OR_OPEN)
+        self.indexer = xapian.TermGenerator()
+        stemmer = xapian.Stem("english")
+        self.indexer.set_stemmer(stemmer)
+
+    def finish_indexing(self):
+        # Ensure the db lock is removed.
+        del self.database
+
+    def add_document(self, path, title, text):
+        self.database.begin_transaction()
+        # sphinx_page_path is used to easily retrieve documents by path.
+        sphinx_page_path = '"sphinxpagepath%s"' % path.replace('/', '_')
+        # Delete the old document if it exists.
+        self.database.delete_document(sphinx_page_path)
+
+        doc = xapian.Document()
+        doc.set_data(text)
+        doc.add_value(self.DOC_PATH, path)
+        doc.add_value(self.DOC_TITLE, title)
+        self.indexer.set_document(doc)
+        self.indexer.index_text(text)
+        doc.add_term(sphinx_page_path)
+        for word in text.split():
+            doc.add_posting(word, 1)
+        self.database.add_document(doc)
+        self.database.commit_transaction()
+
+    def handle_query(self, q):
+        database = xapian.Database(self.db_path)
+        enquire = xapian.Enquire(database)
+        qp = xapian.QueryParser()
+        stemmer = xapian.Stem("english")
+        qp.set_stemmer(stemmer)
+        qp.set_database(database)
+        qp.set_stemming_strategy(xapian.QueryParser.STEM_SOME)
+        query = qp.parse_query(q)
+
+        # Find the top 100 results for the query.
+        enquire.set_query(query)
+        matches = enquire.get_mset(0, 100)
+
+        results = []
+
+        for m in matches:
+            context = self.extract_context(m.document.get_data())
+            results.append((m.document.get_value(self.DOC_PATH),
+                            m.document.get_value(self.DOC_TITLE),
+                            ''.join(context) ))
+
+        return results

sphinx/websupport/storage/__init__.py

@@ -0,0 +1,118 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.websupport.storage
+    ~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    Storage for the websupport package.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+class StorageBackend(object):
+    def pre_build(self):
+        """Called immediately before the build process begins. Use this
+        to prepare the StorageBackend for the addition of nodes.
+        """
+        pass
+
+    def add_node(self, document, line, source):
+        """Add a node to the StorageBackend.
+
+        :param document: the name of the document the node belongs to.
+
+        :param line: the line in the source where the node begins.
+
+        :param source: the source files name.
+        """
+        raise NotImplementedError()
+
+    def post_build(self):
+        """Called after a build has completed. Use this to finalize the
+        addition of nodes if needed.
+        """
+        pass
+
+    def add_comment(self, text, displayed, username, time,
+                    proposal, node_id, parent_id, moderator):
+        """Called when a comment is being added.
+
+        :param text: the text of the comment
+        :param displayed: whether the comment should be displayed
+        :param username: the name of the user adding the comment
+        :param time: a date object with the time the comment was added
+        :param proposal: the text of the proposal the user made
+        :param node_id: the id of the node that the comment is being added to
+        :param parent_id: the id of the comment's parent comment.
+        :param moderator: whether the user adding the comment is a moderator
+        """
+        raise NotImplementedError()
+
+    def delete_comment(self, comment_id, username, moderator):
+        """Delete a comment.
+
+        Raises :class:`~sphinx.websupport.errors.UserNotAuthorizedError`
+        if moderator is False and `username` doesn't match the username
+        on the comment.
+
+        :param comment_id: The id of the comment being deleted.
+        :param username: The username of the user requesting the deletion.
+        :param moderator: Whether the user is a moderator.
+        """
+        raise NotImplementedError()
+
+    def get_metadata(self, docname, moderator):
+        """Get metadata for a document. This is currently just a dict
+        of node_id's with associated comment counts.
+
+        :param docname: the name of the document to get metadata for.
+        :param moderator: whether the requester is a moderator.
+        """
+        raise NotImplementedError()
+
+    def get_data(self, node_id, username, moderator):
+        """Called to retrieve all data for a node. This should return a
+        dict with two keys, *source* and *comments* as described by
+        :class:`~sphinx.websupport.WebSupport`'s
+        :meth:`~sphinx.websupport.WebSupport.get_data` method.
+
+        :param node_id: The id of the node to get data for.
+        :param username: The name of the user requesting the data.
+        :param moderator: Whether the requestor is a moderator.
+        """
+        raise NotImplementedError()
+
+    def process_vote(self, comment_id, username, value):
+        """Process a vote that is being cast. `value` will be either -1, 0,
+        or 1.
+
+        :param comment_id: The id of the comment being voted on.
+        :param username: The username of the user casting the vote.
+        :param value: The value of the vote being cast.
+        """
+        raise NotImplementedError()
+
+    def update_username(self, old_username, new_username):
+        """If a user is allowed to change their username this method should
+        be called so that there is not stagnate data in the storage system.
+
+        :param old_username: The username being changed.
+        :param new_username: What the username is being changed to.
+        """
+        raise NotImplementedError()
+
+    def accept_comment(self, comment_id):
+        """Called when a moderator accepts a comment. After the method is
+        called the comment should be displayed to all users.
+
+        :param comment_id: The id of the comment being accepted.
+        """
+        raise NotImplementedError()
+
+    def reject_comment(self, comment_id):
+        """Called when a moderator rejects a comment. The comment should
+        then be deleted.
+
+        :param comment_id: The id of the comment being accepted.
+        """
+        raise NotImplementedError()

sphinx/websupport/storage/db.py

@@ -0,0 +1,205 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.websupport.storage.db
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    SQLAlchemy table and mapper definitions used by the
+    :class:`sphinx.websupport.comments.SQLAlchemyStorage`.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+from datetime import datetime
+
+from sqlalchemy import Column, Integer, Text, String, Boolean, ForeignKey,\
+                       DateTime
+from sqlalchemy.schema import UniqueConstraint
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm import relation, sessionmaker, aliased
+
+Base = declarative_base()
+
+Session = sessionmaker()
+
+db_prefix = 'sphinx_'
+
+class Node(Base):
+    """Data about a Node in a doctree."""
+    __tablename__ = db_prefix + 'nodes'
+
+    id = Column(Integer, primary_key=True)
+    document = Column(String(256), nullable=False)
+    line = Column(Integer)
+    source = Column(Text, nullable=False)
+
+    def nested_comments(self, username, moderator):
+        """Create a tree of comments. First get all comments that are
+        descendents of this node, then convert them to a tree form.
+
+        :param username: the name of the user to get comments for.
+        :param moderator: whether the user is moderator.
+        """
+        session = Session()
+
+        if username:
+            # If a username is provided, create a subquery to retrieve all
+            # votes by this user. We will outerjoin with the comment query
+            # with this subquery so we have a user's voting information.
+            sq = session.query(CommentVote).\
+                filter(CommentVote.username == username).subquery()
+            cvalias = aliased(CommentVote, sq)
+            q = session.query(Comment, cvalias.value).outerjoin(cvalias)
+        else:
+            # If a username is not provided, we don't need to join with
+            # CommentVote.
+            q = session.query(Comment)
+
+        # Filter out all comments not descending from this node.
+        q = q.filter(Comment.path.like(str(self.id) + '.%'))
+
+        if not moderator:
+            q = q.filter(Comment.displayed == True)
+
+        # Retrieve all results. Results must be ordered by Comment.path
+        # so that we can easily transform them from a flat list to a tree.
+        results = q.order_by(Comment.path).all()
+        session.close()
+
+        return self._nest_comments(results, username)
+
+    def _nest_comments(self, results, username):
+        """Given the flat list of results, convert the list into a
+        tree.
+
+        :param results: the flat list of comments
+        :param username: the name of the user requesting the comments.
+        """
+        comments = []
+        list_stack = [comments]
+        for r in results:
+            comment, vote = r if username else (r, 0)
+
+            inheritance_chain = comment.path.split('.')[1:]
+
+            if len(inheritance_chain) == len(list_stack) + 1:
+                parent = list_stack[-1][-1]
+                list_stack.append(parent['children'])
+            elif len(inheritance_chain) < len(list_stack):
+                while len(inheritance_chain) < len(list_stack):
+                    list_stack.pop()
+
+            list_stack[-1].append(comment.serializable(vote=vote))
+
+        return comments
+
+    def __init__(self, document, line, source):
+        self.document = document
+        self.line = line
+        self.source = source
+
+class Comment(Base):
+    """An individual Comment being stored."""
+    __tablename__ = db_prefix + 'comments'
+
+    id = Column(Integer, primary_key=True)
+    rating = Column(Integer, nullable=False)
+    time = Column(DateTime, nullable=False)
+    text = Column(Text, nullable=False)
+    displayed = Column(Boolean, index=True, default=False)
+    username = Column(String(64))
+    proposal = Column(Text)
+    proposal_diff = Column(Text)
+    path = Column(String(256), index=True)
+
+    node_id = Column(Integer, ForeignKey(db_prefix + 'nodes.id'))
+    node = relation(Node, backref="comments")
+
+    def __init__(self, text, displayed, username, rating, time,
+                 proposal, proposal_diff):
+        self.text = text
+        self.displayed = displayed
+        self.username = username
+        self.rating = rating
+        self.time = time
+        self.proposal = proposal
+        self.proposal_diff = proposal_diff
+
+    def set_path(self, node_id, parent_id):
+        """Set the materialized path for this comment."""
+        # This exists because the path can't be set until the session has
+        # been flushed and this Comment has an id.
+        if node_id:
+            self.node_id = node_id
+            self.path = '%s.%s' % (node_id, self.id)
+        else:
+            session = Session()
+            parent_path = session.query(Comment.path).\
+                filter(Comment.id == parent_id).one().path
+            session.close()
+            self.node_id = parent_path.split('.')[0]
+            self.path = '%s.%s' %  (parent_path, self.id)
+
+    def serializable(self, vote=0):
+        """Creates a serializable representation of the comment. This is
+        converted to JSON, and used on the client side.
+        """
+        delta = datetime.now() - self.time
+
+        time = {'year': self.time.year,
+                'month': self.time.month,
+                'day': self.time.day,
+                'hour': self.time.hour,
+                'minute': self.time.minute,
+                'second': self.time.second,
+                'iso': self.time.isoformat(),
+                'delta': self.pretty_delta(delta)}
+
+        path = self.path.split('.')
+        node = path[0] if len(path) == 2 else None
+        parent = path[-2] if len(path) > 2 else None
+
+        return {'text': self.text,
+                'username': self.username or 'Anonymous',
+                'id': self.id,
+                'node': node,
+                'parent': parent,
+                'rating': self.rating,
+                'displayed': self.displayed,
+                'age': delta.seconds,
+                'time': time,
+                'vote': vote or 0,
+                'proposal_diff': self.proposal_diff,
+                'children': []}
+
+    def pretty_delta(self, delta):
+        """Create a pretty representation of the Comment's age.
+        (e.g. 2 minutes).
+        """
+        days = delta.days
+        seconds = delta.seconds
+        hours = seconds / 3600
+        minutes = seconds / 60
+
+        if days == 0:
+            dt = (minutes, 'minute') if hours == 0 else (hours, 'hour')
+        else:
+            dt = (days, 'day')
+
+        return '%s %s ago' % dt if dt[0] == 1 else '%s %ss ago' % dt
+
+class CommentVote(Base):
+    """A vote a user has made on a Comment."""
+    __tablename__ = db_prefix + 'commentvote'
+
+    username = Column(String(64), primary_key=True)
+    comment_id = Column(Integer, ForeignKey(db_prefix + 'comments.id'),
+                        primary_key=True)
+    comment = relation(Comment, backref="votes")
+    # -1 if downvoted, +1 if upvoted, 0 if voted then unvoted.
+    value = Column(Integer, nullable=False)
+
+    def __init__(self, comment_id, username, value):
+        self.comment_id = comment_id
+        self.username = username
+        self.value = value

sphinx/websupport/storage/differ.py

@@ -0,0 +1,78 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.websupport.storage.differ
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    A differ for creating an HTML representations of proposal diffs
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+import re
+from cgi import escape
+from difflib import Differ
+
+class CombinedHtmlDiff(object):
+    """Create an HTML representation of the differences between two pieces
+    of text.
+    """
+    highlight_regex = re.compile(r'([\+\-\^]+)')
+
+    def make_html(self, source, proposal):
+        """Return the HTML representation of the differences between
+        `source` and `proposal`.
+
+        :param source: the original text
+        :param proposal: the proposed text
+        """
+        proposal = escape(proposal)
+
+        differ = Differ()
+        diff = list(differ.compare(source.splitlines(1),
+                                   proposal.splitlines(1)))
+        html = []
+        line = diff.pop(0)
+        next = diff.pop(0)
+        while True:
+            html.append(self._handle_line(line, next))
+            line = next
+            try:
+                next = diff.pop(0)
+            except IndexError:
+                self._handle_line(line)
+                break
+        return ''.join(html)
+
+    def _handle_line(self, line, next=None):
+        """Handle an individual line in a diff."""
+        prefix = line[0]
+        text = line[2:]
+
+        if prefix == ' ':
+            return text
+        elif prefix == '?':
+            return ''
+
+        if next is not None and next[0] == '?':
+            tag = 'ins' if prefix == '+' else 'del'
+            text = self._highlight_text(text, next, tag)
+        css_class = 'prop_added' if prefix == '+' else 'prop_removed'
+
+        return '<span class="%s">%s</span>\n' % (css_class, text.rstrip())
+
+    def _highlight_text(self, text, next, tag):
+        """Highlight the specific changes made to a line by adding
+        <ins> and <del> tags.
+        """
+        next = next[2:]
+        new_text = []
+        start = 0
+        for match in self.highlight_regex.finditer(next):
+            new_text.append(text[start:match.start()])
+            new_text.append('<%s>' % tag)
+            new_text.append(text[match.start():match.end()])
+            new_text.append('</%s>' % tag)
+            start = match.end()
+        new_text.append(text[start:])
+        return ''.join(new_text)

sphinx/websupport/storage/sqlalchemystorage.py

@@ -0,0 +1,160 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.websupport.storage.sqlalchemystorage
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    An SQLAlchemy storage backend.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+from datetime import datetime
+
+from sqlalchemy.orm import aliased
+from sqlalchemy.sql import func
+
+from sphinx.websupport.errors import *
+from sphinx.websupport.storage import StorageBackend
+from sphinx.websupport.storage.db import Base, Node, Comment, CommentVote,\
+                                          Session
+from sphinx.websupport.storage.differ import CombinedHtmlDiff
+
+class SQLAlchemyStorage(StorageBackend):
+    """A :class:`~sphinx.websupport.storage.StorageBackend` using
+    SQLAlchemy.
+    """
+    def __init__(self, engine):
+        self.engine = engine
+        Base.metadata.bind = engine
+        Base.metadata.create_all()
+        Session.configure(bind=engine)
+
+    def pre_build(self):
+        self.build_session = Session()
+
+    def add_node(self, document, line, source):
+        node = Node(document, line, source)
+        self.build_session.add(node)
+        self.build_session.flush()
+        return node
+
+    def post_build(self):
+        self.build_session.commit()
+        self.build_session.close()
+
+    def add_comment(self, text, displayed, username, time,
+                    proposal, node_id, parent_id, moderator):
+        session = Session()
+        proposal_diff = None
+
+        if node_id and proposal:
+            node = session.query(Node).filter(Node.id == node_id).one()
+            differ = CombinedHtmlDiff()
+            proposal_diff = differ.make_html(node.source, proposal)
+        elif parent_id:
+            parent = session.query(Comment.displayed).\
+                filter(Comment.id == parent_id).one()
+            if not parent.displayed:
+                raise CommentNotAllowedError(
+                    "Can't add child to a parent that is not displayed")
+
+        comment = Comment(text, displayed, username, 0,
+                          time or datetime.now(), proposal, proposal_diff)
+        session.add(comment)
+        session.flush()
+        # We have to flush the session before setting the path so the
+        # Comment has an id.
+        comment.set_path(node_id, parent_id)
+        session.commit()
+        d = comment.serializable()
+        session.close()
+        return d
+
+    def delete_comment(self, comment_id, username, moderator):
+        session = Session()
+        comment = session.query(Comment).\
+            filter(Comment.id == comment_id).one()
+        if moderator or comment.username == username:
+            comment.username = '[deleted]'
+            comment.text = '[deleted]'
+            comment.proposal = ''
+            session.commit()
+            session.close()
+        else:
+            session.close()
+            raise UserNotAuthorizedError()
+
+    def get_metadata(self, docname, moderator):
+        session = Session()
+        subquery = session.query(
+            Comment.id, Comment.node_id,
+            func.count('*').label('comment_count')).group_by(
+            Comment.node_id).subquery()
+        nodes = session.query(Node.id, subquery.c.comment_count).outerjoin(
+            (subquery, Node.id==subquery.c.node_id)).filter(
+            Node.document==docname)
+        session.close()
+        session.commit()
+        return dict([(k, v or 0) for k, v in nodes])
+
+    def get_data(self, node_id, username, moderator):
+        session = Session()
+        node = session.query(Node).filter(Node.id == node_id).one()
+        session.close()
+        comments = node.nested_comments(username, moderator)
+        return {'source': node.source,
+                'comments': comments}
+
+    def process_vote(self, comment_id, username, value):
+        session = Session()
+
+        subquery = session.query(CommentVote).filter(
+            CommentVote.username == username).subquery()
+        vote_alias = aliased(CommentVote, subquery)
+        q = session.query(Comment, vote_alias).outerjoin(vote_alias).filter(
+            Comment.id == comment_id)
+        comment, vote = q.one()
+
+        if vote is None:
+            vote = CommentVote(comment_id, username, value)
+            comment.rating += value
+        else:
+            comment.rating += value - vote.value
+            vote.value = value
+
+        session.add(vote)
+        session.commit()
+        session.close()
+
+    def update_username(self, old_username, new_username):
+        session = Session()
+
+        session.query(Comment).filter(Comment.username == old_username).\
+            update({Comment.username: new_username})
+        session.query(CommentVote).\
+            filter(CommentVote.username == old_username).\
+            update({CommentVote.username: new_username})
+
+        session.commit()
+        session.close()
+
+    def accept_comment(self, comment_id):
+        session = Session()
+
+        comment = session.query(Comment).filter(
+            Comment.id == comment_id).update(
+            {Comment.displayed: True})
+
+        session.commit()
+        session.close()
+
+    def reject_comment(self, comment_id):
+        session = Session()
+
+        comment = session.query(Comment).\
+            filter(Comment.id == comment_id).one()
+        session.delete(comment)
+
+        session.commit()
+        session.close()

sphinx/writers/websupport.py

@@ -0,0 +1,61 @@
+# -*- coding: utf-8 -*-
+"""
+    sphinx.writers.websupport
+    ~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    docutils writers handling Sphinx' custom nodes.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+from sphinx.writers.html import HTMLTranslator
+
+class WebSupportTranslator(HTMLTranslator):
+    """
+    Our custom HTML translator.
+    """
+    commentable_nodes = ['paragraph']
+
+    def __init__(self, builder, *args, **kwargs):
+        HTMLTranslator.__init__(self, builder, *args, **kwargs)
+        self.comment_class = 'spxcmt'
+        self.init_support()
+
+    def init_support(self):
+        self.cur_node = None
+
+    def dispatch_visit(self, node):
+        if node.__class__.__name__ in self.commentable_nodes:
+            self.handle_visit_commentable(node)
+        HTMLTranslator.dispatch_visit(self, node)
+
+    def dispatch_departure(self, node):
+        HTMLTranslator.dispatch_departure(self, node)
+        if node.__class__.__name__ in self.commentable_nodes:
+            self.handle_depart_commentable(node)
+
+    def handle_visit_commentable(self, node):
+        # If this node is nested inside another commentable node this
+        # node will not be commented.
+        if self.cur_node is None:
+            self.cur_node = self.add_db_node(node)
+            # We will place the node in the HTML id attribute. If the node
+            # already has an id (for indexing purposes) put an empty
+            # span with the existing id directly before this node's HTML.
+            if node.attributes['ids']:
+                self.body.append('<span id="%s"></span>'
+                                 % node.attributes['ids'][0])
+            node.attributes['ids'] = ['s%s' % self.cur_node.id]
+            node.attributes['classes'].append(self.comment_class)
+
+    def handle_depart_commentable(self, node):
+        if self.comment_class in node.attributes['classes']:
+            self.cur_node = None
+
+    def add_db_node(self, node):
+        storage = self.builder.app.storage
+        db_node_id = storage.add_node(document=self.builder.cur_docname,
+                                      line=node.line,
+                                      source=node.rawsource or node.astext())
+        return db_node_id

tests/test_searchadapters.py

@@ -0,0 +1,83 @@
+# -*- coding: utf-8 -*-
+"""
+    test_searchadapters
+    ~~~~~~~~~~~~~~~~~~~
+
+    Test the Web Support Package search adapters.
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+import os, sys
+from StringIO import StringIO
+
+from util import *
+from sphinx.websupport import WebSupport
+
+
+def clear_builddir():
+    (test_root / 'websupport').rmtree(True)
+
+
+def teardown_module():
+    (test_root / 'generated').rmtree(True)
+    clear_builddir()
+
+
+def search_adapter_helper(adapter):
+    clear_builddir()
+
+    settings = {'builddir': os.path.join(test_root, 'websupport'),
+                'status': StringIO(),
+                'warning': StringIO()}
+    settings.update({'srcdir': test_root,
+                     'search': adapter})
+    support = WebSupport(**settings)
+    support.build()
+
+    s = support.search
+
+    # Test the adapters query method. A search for "Epigraph" should return
+    # one result.
+    results = s.query(u'Epigraph')
+    assert len(results) == 1, \
+        '%s search adapter returned %s search result(s), should have been 1'\
+        % (adapter, len(results))
+
+    # Make sure documents are properly updated by the search adapter.
+    s.init_indexing(changed=['markup'])
+    s.add_document(u'markup', u'title', u'SomeLongRandomWord')
+    s.finish_indexing()
+    # Now a search for "Epigraph" should return zero results.
+    results = s.query(u'Epigraph')
+    assert len(results) == 0, \
+        '%s search adapter returned %s search result(s), should have been 0'\
+        % (adapter, len(results))
+    # A search for "SomeLongRandomWord" should return one result.
+    results = s.query(u'SomeLongRandomWord')
+    assert len(results) == 1, \
+        '%s search adapter returned %s search result(s), should have been 1'\
+        % (adapter, len(results))
+    # Make sure it works through the WebSupport API
+    html = support.get_search_results(u'SomeLongRandomWord')
+
+
+def test_xapian():
+    # Don't run tests if xapian is not installed.
+    try:
+        import xapian
+        search_adapter_helper('xapian')
+    except ImportError:
+        sys.stderr.write('info: not running xapian tests, ' \
+                         'xapian doesn\'t seem to be installed')
+
+
+def test_whoosh():
+    # Don't run tests if whoosh is not installed.
+    try:
+        import whoosh
+        search_adapter_helper('whoosh')
+    except ImportError:
+        sys.stderr.write('info: not running whoosh tests, ' \
+                         'whoosh doesn\'t seem to be installed')

tests/test_websupport.py

@@ -0,0 +1,254 @@
+# -*- coding: utf-8 -*-
+"""
+    test_websupport
+    ~~~~~~~~~~~~~~~
+
+    Test the Web Support Package
+
+    :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+    :license: BSD, see LICENSE for details.
+"""
+
+import os
+from StringIO import StringIO
+
+from sphinx.websupport import WebSupport
+from sphinx.websupport.errors import *
+from sphinx.websupport.storage.differ import CombinedHtmlDiff
+from sphinx.websupport.storage.sqlalchemystorage import Session, \
+    SQLAlchemyStorage, Comment, CommentVote
+from sphinx.websupport.storage.db import Node
+from util import *
+
+try:
+    from functools import wraps
+except ImportError:
+    # functools is new in 2.4
+    wraps = lambda f: (lambda w: w)
+
+
+default_settings = {'builddir': os.path.join(test_root, 'websupport'),
+                    'status': StringIO(),
+                    'warning': StringIO()}
+
+def teardown_module():
+    (test_root / 'generated').rmtree(True)
+    (test_root / 'websupport').rmtree(True)
+
+
+def with_support(*args, **kwargs):
+    """Make a WebSupport object and pass it the test."""
+    settings = default_settings.copy()
+    settings.update(kwargs)
+
+    def generator(func):
+        @wraps(func)
+        def new_func(*args2, **kwargs2):
+            support = WebSupport(**settings)
+            func(support, *args2, **kwargs2)
+        return new_func
+    return generator
+
+
+@with_support()
+def test_no_srcdir(support):
+    """Make sure the correct exception is raised if srcdir is not given."""
+    raises(SrcdirNotSpecifiedError, support.build)
+
+
+@with_support(srcdir=test_root)
+def test_build(support):
+    support.build()
+
+
+@with_support()
+def test_get_document(support):
+    raises(DocumentNotFoundError, support.get_document, 'nonexisting')
+
+    contents = support.get_document('contents')
+    assert contents['title'] and contents['body'] \
+        and contents['sidebar'] and contents['relbar']
+
+
+@with_support()
+def test_comments(support):
+    session = Session()
+    nodes = session.query(Node).all()
+    first_node = nodes[0]
+    second_node = nodes[1]
+
+    # Create a displayed comment and a non displayed comment.
+    comment = support.add_comment('First test comment',
+                                  node_id=str(first_node.id),
+                                  username='user_one')
+    hidden_comment = support.add_comment('Hidden comment',
+                                         node_id=str(first_node.id),
+                                         displayed=False)
+    # Make sure that comments can't be added to a comment where
+    # displayed == False, since it could break the algorithm that
+    # converts a nodes comments to a tree.
+    raises(CommentNotAllowedError, support.add_comment, 'Not allowed',
+           parent_id=str(hidden_comment['id']))
+    # Add a displayed and not displayed child to the displayed comment.
+    support.add_comment('Child test comment', parent_id=str(comment['id']),
+                        username='user_one')
+    support.add_comment('Hidden child test comment',
+                        parent_id=str(comment['id']), displayed=False)
+    # Add a comment to another node to make sure it isn't returned later.
+    support.add_comment('Second test comment',
+                        node_id=str(second_node.id),
+                        username='user_two')
+
+    # Access the comments as a moderator.
+    data = support.get_data(str(first_node.id), moderator=True)
+    comments = data['comments']
+    children = comments[0]['children']
+    assert len(comments) == 2
+    assert comments[1]['text'] == 'Hidden comment'
+    assert len(children) == 2
+    assert children[1]['text'] == 'Hidden child test comment'
+
+    # Access the comments without being a moderator.
+    data = support.get_data(str(first_node.id))
+    comments = data['comments']
+    children = comments[0]['children']
+    assert len(comments) == 1
+    assert comments[0]['text'] == 'First test comment'
+    assert len(children) == 1
+    assert children[0]['text'] == 'Child test comment'
+
+
+@with_support()
+def test_voting(support):
+    session = Session()
+    nodes = session.query(Node).all()
+    node = nodes[0]
+
+    comment = support.get_data(str(node.id))['comments'][0]
+
+    def check_rating(val):
+        data = support.get_data(str(node.id))
+        comment = data['comments'][0]
+        assert comment['rating'] == val, '%s != %s' % (comment['rating'], val)
+
+    support.process_vote(comment['id'], 'user_one', '1')
+    support.process_vote(comment['id'], 'user_two', '1')
+    support.process_vote(comment['id'], 'user_three', '1')
+    check_rating(3)
+    support.process_vote(comment['id'], 'user_one', '-1')
+    check_rating(1)
+    support.process_vote(comment['id'], 'user_one', '0')
+    check_rating(2)
+
+    # Make sure a vote with value > 1 or < -1 can't be cast.
+    raises(ValueError, support.process_vote, comment['id'], 'user_one', '2')
+    raises(ValueError, support.process_vote, comment['id'], 'user_one', '-2')
+
+    # Make sure past voting data is associated with comments when they are
+    # fetched.
+    data = support.get_data(str(node.id), username='user_two')
+    comment = data['comments'][0]
+    assert comment['vote'] == 1, '%s != 1' % comment['vote']
+
+
+@with_support()
+def test_proposals(support):
+    session = Session()
+    node = session.query(Node).first()
+
+    data = support.get_data(str(node.id))
+
+    source = data['source']
+    proposal = source[:5] + source[10:15] + 'asdf' + source[15:]
+
+    comment = support.add_comment('Proposal comment',
+                                  node_id=str(node.id),
+                                  proposal=proposal)
+
+
+@with_support()
+def test_user_delete_comments(support):
+    def get_comment():
+        session = Session()
+        node = session.query(Node).first()
+        session.close()
+        return support.get_data(str(node.id))['comments'][0]
+
+    comment = get_comment()
+    assert comment['username'] == 'user_one'
+    # Make sure other normal users can't delete someone elses comments.
+    raises(UserNotAuthorizedError, support.delete_comment,
+           comment['id'], username='user_two')
+    # Now delete the comment using the correct username.
+    support.delete_comment(comment['id'], username='user_one')
+    comment = get_comment()
+    assert comment['username'] == '[deleted]'
+    assert comment['text'] == '[deleted]'
+
+
+@with_support()
+def test_moderator_delete_comments(support):
+    def get_comment():
+        session = Session()
+        node = session.query(Node).first()
+        session.close()
+        return support.get_data(str(node.id), moderator=True)['comments'][1]
+
+    comment = get_comment()
+    support.delete_comment(comment['id'], username='user_two',
+                           moderator=True)
+    comment = get_comment()
+    assert comment['username'] == '[deleted]'
+    assert comment['text'] == '[deleted]'
+
+
+@with_support()
+def test_update_username(support):
+    support.update_username('user_two', 'new_user_two')
+    session = Session()
+    comments = session.query(Comment).\
+        filter(Comment.username == 'user_two').all()
+    assert len(comments) == 0
+    votes = session.query(CommentVote).\
+        filter(CommentVote.username == 'user_two')
+    assert len(comments) == 0
+    comments = session.query(Comment).\
+        filter(Comment.username == 'new_user_two').all()
+    assert len(comments) == 1
+    votes = session.query(CommentVote).\
+        filter(CommentVote.username == 'new_user_two')
+    assert len(comments) == 1
+
+
+called = False
+def moderation_callback(comment):
+    global called
+    called = True
+
+
+@with_support(moderation_callback=moderation_callback)
+def test_moderation(support):
+    accepted = support.add_comment('Accepted Comment', node_id=3,
+                                   displayed=False)
+    rejected = support.add_comment('Rejected comment', node_id=3,
+                                   displayed=False)
+    # Make sure the moderation_callback is called.
+    assert called == True
+    # Make sure the user must be a moderator.
+    raises(UserNotAuthorizedError, support.accept_comment, accepted['id'])
+    raises(UserNotAuthorizedError, support.reject_comment, accepted['id'])
+    support.accept_comment(accepted['id'], moderator=True)
+    support.reject_comment(rejected['id'], moderator=True)
+    comments = support.get_data(3)['comments']
+    assert len(comments) == 1
+    comments = support.get_data(3, moderator=True)['comments']
+    assert len(comments) == 1
+
+
+def test_differ():
+    differ = CombinedHtmlDiff()
+    source = 'Lorem ipsum dolor sit amet,\nconsectetur adipisicing elit,\n' \
+        'sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.'
+    prop = 'Lorem dolor sit amet,\nconsectetur nihil adipisicing elit,\n' \
+        'sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.'
+    differ.make_html(source, prop)