Rendering ReST with Klein and Twisted Templates

In a previous life, I spent about 25 hours a day worrying about content management systems written in Python. As a result of the battle scars built up during those days, I have developed a pretty strong aversion for a heavy CMS when a simple approach will do. Especially if the users are technologically proficient.

At MindPool, we're building out our infrastructure right now using Twisted so that we can take advantage of the super amazing numbers of protocols that Twisted supports to provide some pretty unique combined services for our customers (among the many other types of services we are providing). For our website, we're using the Bottle/Flask-inspired Klein as our micro web framework, and this uses the most excellent Twisted templating. (We are, of course, also using Twitter Bootstrap.)

Here's the rub, though: we want to manage our content in the git repo for our site with ReStructured Text files, and there's no way to tell the template rendering machinery (the flattener code) to allow raw HTML into the mix. As such, my first attempt at ReST support was rendering HTML tags all over the user-facing content.

This ended up being a blessing in disguise, though, as I was fairly unhappy with the third-party dependencies that had popped up as a result of getting this to work. After a couple false starts, I was hot on the trail of a good solution: convert the docutils-generated HTML (from the ReST source files) to Twisted Stan tags, and push those into the renderers.

This ended up working like a champ. Here's what I did:

1. Created a couple of utility functions for easily getting HTML from ReST and Stan from ReST.

	from docutils.core import publish_string as render_rst
	from twisted.web.template import Tag, XMLString
	def rstToHTML(rst):
	overrides = {
	'output_encoding': 'unicode',
	'input_encoding': 'unicode'
	}
	return render_rst(
	rst.decode("utf-8"), writer_name="html",
	settings_overrides=overrides)
	def checkTag(tag):
	if isinstance(tag, basestring):
	return False
	return True
	def rstToStan(rst):
	html = rstToHTML(rst)
	# fix a bad encoding in docutils
	html = html.replace('encoding="unicode"', '')
	stan = XMLString(html).load()[0]
	# we're always going to get the same structure back, at least
	# for the installed version of docutils, so let's trip out the
	# crap that we don't need (also, let's safeguard against future
	# changes)
	if stan.tagName != "html":
	raise ValueError("Unexpected top-level HTML tag.")
	# let's ditch the children whose sole contents are "\n" strings
	head, body = [x for x in stan.children if checkTag(x)]
	if head.tagName != "head":
	raise ValueError("Expected 'head' node, got '%s'" % (
	head.tagName))
	if body.tagName != "body":
	raise ValueError("Expected 'body' node, got '%s'" % (
	body.tagName))
	# by just returning the contents of the body tag, we're avoiding
	# all the CSS and, in fact, the complete HTML file that the
	# docutils ReST renderer provides by default
	return body.children

view raw utils.py hosted with ❤ by GitHub
2. Wrote a custom IRenderable for ReST content (not strictly necessary, but organizationally useful, given what else will be added in the future).

	from twisted.web.iweb import IRenderable
	from zope.interface import implements
	class ReSTContent(object):
	"""
	Though not strictly necessary as a dedicated class, I wanted to
	put this code here to assist with organization of content type
	renderers, as this will not be the only one.
	"""
	implements(IRenderable)
	def __init__(self, rstData):
	self.rstData = rstData
	def render(self, request):
	return utils.rstToStan(self.rstData)

view raw renderer.py hosted with ❤ by GitHub
3. Updated the base class for "content" page templates to dispatch, depending upon content type.

	import const, fragments, renderer
	class ContentPage(BasePage):
	"""
	Note that the BasePage class is not included in this
	example. It and its parent class manage template
	loading, slot-filling, etc.
	"""
	contentType = const.contentTypes["rst"]
	contentData = ""
	# Hobo caching
	_cachedContent = ""
	def renderContentData(self):
	if not self._cachedContent:
	if self.contentType == const.contentTypes["rst"]:
	self._cachedContent = renderer.ReSTContent(
	self.contentData)
	elif self.contentType == const.contentTypes["html"]:
	self._cachedContent = self.contentData
	return self._cachedContent
	@renderer
	def topnav(self, request, tag):
	return fragments.TopNavFragment()
	@renderer
	def contentarea(self, request, tag):
	return fragments.ContentFragment(self.renderContentData())

view raw basepages.py hosted with ❤ by GitHub

Afterwards I was rewarded with some nicely rendered content on the staging MindPool site :-) (once the content text has been completed, we'll be pushing it live).

Kudos to David Reid for Klein and (as usual) to the Twisted community for one hell of a framework that is the engine of my internet.