Infogami Developer Tutorial

This document is not maintained.

The Open Library interface is powered by infogami -- a cleaner, simpler alternative to other wiki applications. But unlike other wikis, Infogami has the flexibility to handle different classes of data. Most wikis let you store unstructured pages -- big blocks of text. Infogami lets you store structured data.  [1]

In addition to this, infogami facilitates the creation of dynamic HTML templates and macros. This flexible environment enables users to create, share and build collaborative interfaces. With Open Library in particular, we are focused on building a productive and vital community focused on the discovery of books.

Applications are written by extending Infogami through two layers: plugins and templates. Plugins are Python modules that get loaded into Infogami through a special API. (See [an overview of Infogami plugins][6].) They are invoked by submitting HTTP requests to the application, either HTML form posts or direct GET requests. Plugins can use any library or application code that they wish, and they create Python objects to represent results, that then get expanded to HTML by templates. Templates are a mixture of HTML text and user-written code, in the spirit of PHP templates. The user-written code is in a special-purpose scripting language that is approximately a Python subset, which runs in a hopefully-secure server-side interpreter embedded in the Python app that has limited access to system functions and resources.

In this document, you'll learn how to develop for infogami, including building new templates for displaying your own data, running your own copy, and developing new features and plugins.

Summary (Audience Statement)

This document describes the internal workings of the Open Library software, from a developers' point of view. You should read it if you are:

1) a software developer wanting to add new features to Open Library. To do this, you will also have to be a good Python programmer experienced in writing web server applications (not necessarily in Python). The document will explain the Open Library's software architecture and its internal interfaces, and will explain how to write extensions (plugins), but the sections about plugin writing will assume that you are familiar with Python and web programming in general.

If you do not yet know Python, you should first study the Python documentation or the free book Dive into Python. Python is an easy language to learn, but the OL codebase is probably not understandable by complete beginners.
For web server principles, the somewhat dated Philip and Alex's Guide to Web Publishing is still an informative read, though maybe someone can suggest something newer. You should also understand the principles of software security -- see David A Wheeler's page for many documents.

2) A user or web designer wanting to improve or customize the Open Library's user interface, either for yourself or for our whole community. You will mainly want to study the section about template programming. You will need to know how to write HTML and it will help if you've done some server-side template programming (such as in PHP). It will also help if you've had some exposure to Python, but the programming skills you'll need for template writing are generally less intense than they'd be for extension writing.

3) A general user just wanting to know how the software works behind the scenes. You might not understand all the details, but reading the doc should give you a general impression of how sites like this are put together.

4) A librarian or metadata maintainer wanting to process large volumes of metadata for import into the Open Library. If you only want to import a few books, it's probably easiest to use the web interface (or the Z39.50 interface once we have one). To import bulk data, you'll have to process it into a format that Open Library can understand, which may require programming, but you can use your own choices of languages and platforms for that purpose since you only have to create uploadable files, rather than use language-specific interfaces. You'll mainly want to look at the section about data formats and schemas.

If you just want to be an OL user accessing or editing book data, you do NOT need to read this doc. The doc is about how to customize and extend the software, not how to use it. As developers and designers, our goal is to make the site self-explanatory for users and not need much separate documentation, but we do have some user docs at /help.

Introduction

Infogami is a wiki application framework built on web.py. Actual applications (like Open Library) are written by extending Infogami through two layers: plugins and templates. Plugins are Python modules that get loaded into Infogami through a special API. They are invoked by submitting HTTP requests to the application, either HTML form posts or direct GET requests. Plugins can use any library or application code that they wish, and they create Python objects to represent results, that then get expanded to HTML by templates. Templates are a mixture of HTML text and user-written code, approximately in the spirit of PHP templates. The user-written code is in a special-purpose scripting language that is approximately a Python subset, which runs in a hopefully-secure server-side interpreter (embedded in the Python app) that has limited access to system functions and resources.

  Architecture

1.  Web Server

The lighttpd HTTP server runs Infogami through a FastCGI interface using Flup. (There can be multiple concurrent infogami instances that the lighttpd server distributes requests between, although Open Library currently just runs one.) Infogami is written in Python (Open Library currently requires 2.5 or greater) and uses web.py and ThingDB. ThingDB uses PostgreSQL as its data store. Psycopg2 is the Python driver for PostgreSQL. Open Library uses supervise (see also daemontools) to make sure everything keeps running. [2]

2.  Templates

The infogami application relies on various Web templates (these are code+HTML snippets). The initial templates are static files but they get edited through the wiki interface, and new ones get added through the wiki, so the real versions live entirely in the database.

2.1  web.py

The goal of web.py is to build the ideal way to make web apps. web.py allows you to build HTTP responses, makes the database easier to use, and creates a templating evironment that tries to bring Python into HTML. [*]

2.2  infogami

Each infogami page (i.e. something with a URL) has an associated type. Each type contains a schema that states what fields can be used with it and what format those fields are in. Those are used to generate view and edit templates which can then be further customized as a particular type requires.

2.3  OL application

[ See State of the UI ] ...  [3]

3.  Search

Infogami also accepts plug-ins and Open Library uses one for the Solr search engine. Solr is a JSP currently sitting in a Jetty http server, so it communicates with Infogami through a local http socket. Solr itself wraps the Lucene search library. These run under Java (Open Library currently uses Java 1.5). Solr is built under Apache Ant and has a few config and schema files, plus a startup script (solr.sh) that has to be manually edited to set the port number. I think we currently use Lucene as a downloaded .jar file so we don't build it.

The solr-infogami plugin also calls out to an archive.org PHP script that expands basic search queries to advanced queries. It may also start using the openlibrary.org flipbook (with some possible customizations) to display OCA scans for pages containing fulltext search results.

4.  Data

Open Library has a bunch of catalog data and fulltext acquired from various sources, either sitting in the Archive or to be uploaded to there. I think the acquisition processes (including web crawling scripts for some of the data) is outside the scope of an Open Library software install. There are a bunch of additional scripts to make the stuff usable in openlibrary and these need to be documented. These include TDB Conversion Scripts written by dbg, and (for OCA fulltext) Archive Spidering and Solr Importing scripts written by phr.

  Template Basics

5.  Types

There are two kinds of types in infogami. Primitive types and compound types. Primitive types are for representing integers, strings etc.

Other primitive and compound types can be added by creating a page in the wiki with type type/type.

Sometimes you might want to link two different types. For example, book type has an author property and you want to add a books field to the author, which contains all books where author property of the book is this author. This is achieved by adding a backreference to the author type.

5.1  Primitive Types

Available primitive types are:

type/int
type/boolean
type/string - for single line text
type/text - for multiline text

5.2  Compound Types

Compound types specify what properties a thing of that type must have. Available compound types are:

type/type
Type for representing all types. It defines the following properties:

• description of type type/text - description of type
• is_multiple of type type/boolean - to distinguish between primitive and compound types
• properties of type type/property - set of properties a thing of this type can have. This property can have multiple values.
• backreferenes of type type/backreference - set of backreferences a thing of this type can have. This property can have multiple values. Back-references are explained later in this document.

type/property
Type for representing properties in a type. It defines the following properties ...

• name of type type/string - name of the property
• type of type type/type - expected type of the value
• is_unique of type type/boolean - can this type have multiple values?
• description of type type/string - readable description of this property

type/backreference
later ...

type/page
Type for wiki pages. It defines the following properties ...

• title of type type/string - title of the page
• body of type type/text - body of the page

type/template
Type for template pages. It defines the following properties ...

• title of type type/string - title of the template
• body of type type/text - body of the template

6.  Templates

Infogami uses templates to control the look and feel of the site. Custom templates can be provided in the wiki to enhance functionality or customize the look and feel of the site.

There are two kinds of templates. Regular templates and type-specific templates. Type-specific templates are used to render a wiki page based on its type.

Regular templates must be placed at path templates/ and end with .tmpl extension. type-specific templates must be placed at type/ and end with .tmpl extension.

With Open Library, only site administrators can modify the templates that reside in the templates/ and type/ directories. If users want to override these templates, they must create their own set of templates in their user directory, and change their template root in their user preferences. For more information, see Custom Templates.

6.1  Type-specific Templates

There can be 4 type-specific templates for every type.

• view.tmpl - Template to specify how to view a page
• edit.tmpl - Template to specify how to edit a page
• repr.tmpl - Template to specify how objects of this type appear when viewing pages of other types which has a property of this type.
• input.tmpl - Template to specify how objects of this type appear when editing pages of other types which has a property of this type.

For more information on how the repr and input templates are used, please see Useful Functions.

6.2  Default Templates

It is not compulsory to specify all these templates for every type. When a type-specific template is not specified for a type, a default template is used. The following are the default templates:

• templates/default_view.tmpl - View Code or View Page Using Template
• templates/default_edit.tmpl - View Code or View Page Using Template
• templates/default_repr.tmpl - View Code
• templates/default_input.tmpl - View Code
• Note to Anand: what about http://www.openlibrary.org:8080/templates/default_ref.tmpl ?

There are many other interesting regular templates. Some of them are:

• templates/site.tmpl - to specify the overall look and feel of a web site - View Code or View Page Using Template
• templates/diff.tmpl - to display a diff between two versions of a page - View Code or View Page Using Template
• ...

6.3  Custom Templates

A user can create their own set of templates by creating a subdirectory in the 'user' directory ('user/[username]'). A user can create a new template for each set of templates they want to override. If a template does not exist in a user's subdirectory, then the default set of templates will be used. Templates are created in the same manner as any new page is created in a wiki.

For purposes of illustration, we will view the contents of ('user/brewster/*')

• /user/brewster/type/
• /user/brewster/templates/

To view the Open Library using a set of custom templates, you must change your template preferences:

• Log in with your username and password
• Click on "Preferences" in the upper-right corner of the screen
• Change your template root to "/user/[username]"

NOTE: Changing the template root to 'user/brewster' will load the templates used in this example.

It is also possible to create custom Cascading Style Sheet (CSS) files and JavaScript files in the wiki.

To create a custom CSS file in the wiki, create a page in your user directory entitled 'user/[username]/[filename].css' and select 'rawtext' for the pagetype. Once the CSS file has been successfully created, it can be linked to the template using the '$add_stylesheet' command:

JavaScript files can be created in the same manner, using a '.js' extension and the '$add_javascript' command:

It is important to note that when a page with the type 'rawtext' is added to the wiki, the editing function is not readily apparent. To edit a rawtext page, simply add '?m=edit' to the url.

Example:

• http://openlibrary.org/user/anand/master.css
• http://openlibrary.org/user/anand/master.css?m=edit

6.4  Useful Functions

There are some useful functions, which are very handy in writing view and edit templates.

• thingrepr(value, type) - This calls repr template of the specified type with value as argument.
• thinginput(type, name, value) - This calls input template of the specified type with value, name as arguments.

Lets take an example. Suppose there are many types, which need to display an image. Probably they will do that by defining a type type/image for representing images. Instead of worrying how to display and edit (upload) images, these types can just use thingrepr and thinginput macros and define repr and input templates for type/image. Later in future, if you want to change how to display/upload image, you don't have to change in multiple places.

There are some global functions, which are accessible by every template. For more information see Infogami API > Decorators.

7.  Macros

Macros are like functions in any programming language. Macros look very much like templates, but they can be called from wikipages and other templates and macros. Macros can be written in the wiki just like templates. Macros must be placed in macros/ path and type must be type/macro. As a convention, they are named in CamelCase.

7.1  Best Practices

Macros introduce an element of modularity when designing web templates in infogami. A common practice is to use macros for any repeatable design elements. This can be as simple as an HTML callout box, or a script. For instance, the "SAVE", "PREVIEW", and "DELETE" buttons on every edit form on the Open Library site are generated by the "EditButtons" Macro code below:

Intelligent use of macros results in clean, readable code that can be easily understood and repurposed by other users. For instance, the author view template is a good example of a template with a highly modular and streamlined design. View Page Using Template

7.2  Embedding

Macros can be invoked either through the wiki interface when editing a page in Markdown, or as a script when building web templates.

The syntax to call a macro from a wiki page is:

The syntax to call a macro from templates and macros is:

  WikiLanguage

8  Formatting Conventions

8.1  Page Headers

The edit and view templates are defined by two lines of code that must be at the top of the template file in order for them to function properly (The absence of either of these two lines will generate an error). For instance, this example is from the default edition template:

Additional variables can be controlled in these commands. For instance, in an edit template, the preview function can be disabled as follows:

Display of the <TITLE> of the page can be constructed by making a call to a universal language file. For instance:

would display "edit Tom Sawyer" in the edit view of an edition page by accessing the verbiage in the language file for '$_.EDIT' (edit) and the 'title' key in the 'edition' type [ See http://www.openlibrary.org/type/edition/].

and:

would display "View Mark Twain" when viewing an author page by accessing the 'name' attribute in the 'author' type [See http://openlibrary.org/type/author/].

8.2  Format Commands

8.3  Working with Types

The page 'get' statements make a call to the document type to pull the corresponding information for display on the page. For instance the following table illustrates our current author type:

description
is_primitive
properties	website of type type/string
properties	bio of type type/string
properties	photograph of type type/image
properties	name of type type/string
properties	death_date of type type/string
properties	alt_names of type type/string
properties	location of type type/string
properties	birth_date of type type/string
backreferences	Back-reference to authors property of type/edition

Therefore using this command:

on the view template will display the author name wherever used on an author template.

9  Template Syntax

9.1  Conditionals

'If' statements are used in much the same manner as in other scripting languages, invoking an action if a certain state exists. For instance, if page preview is activated in an edit template as follows:

the following command would format the HTML contents of the page wherever it is inserted in the edit template:

The 'if' statement can also be used in conjunction with the 'get' statement to hide empty data fields on a page. For instance:

would display the language a book was published in if that data is available for that book, or:

would display the date of birth and date of death for a given author on the author view template if they exist.

9.2  Loops and Iterations

9.3  Regular Expressions

  The Infogami API

10.  Plugins

Functionality of infogami can be extended by adding new plugins to it. Plugin in infogami is a python package with the following directory structure.

• foo/
• foo/__init__.py - required to make foo a python package
• foo/code.py - this is imported when infogami starts
• foo/templates/ - place to keep templates. This is optional.

see sample_run.py in infogami repository to find out how to add a plugin to infogami.

10.1  Special Pages and Modes

A plugin can add special pages to infogami and add new modes to wiki pages. Special pages are paths which are treated differently than regular wiki pages. login, logout, sitepreferences etc. are some examples of special pages. Modes are actions that be done on a wiki page. view, edit, history, diff etc. are some examples of wiki modes.

infogami.utils.delegate module provides two special classes page and mode. Any class that extends from page becomes a special page with path same as the class name and any class that extends from mode becomes a wiki mode with name of the mode same as the class name.

An example:

 # hello/code.py
 class hellopage(delegate.page)
     def GET(self, site)
         return "hello page!"
class hello(delegate.mode)
     def GET(self, site, path)
         return "hello mode for " + path

If this hello plugin is added to infogami, /hellopage page displays hello page! and /foo?m=hello displays hello mode for foo.

10.2  Actions

A plugin can also add new actions to the system. Action is a utility function, which can be invoked from the command line as shown in the following example.

The default action is startserver and that is invoked when no action is specified.

To convert a function in to an action, it must decorated with infogami.action decorator. The following is an example of action.

Once this plugin is added to infogami, this action can be used.

The help action prints the list of all available actions.

There are some important actions, that every plugin author must know.

install: The install action runs all the install hooks. Install hook is a function without any arguments decorated by infogami.install_hook. Any plugin can add a new install hook to do some actions to initialize an infogami instance. Some of the actions are install hooks too.

movefiles: The movefiles action moves all the files in $plugin/files/ from every plugin to static/ directory. This is also an install hook. If a plugin needs to access any static resources, they must be put in files/ directory of that plugin. Keeping resources directly in static/ is not a good idea and should be discouraged.

movetemplates: The movetemplates action moves templates from every plugin to wiki. Optional prefix can be specified to limit the templates that are moved into the wiki. This is also an install hook.

10.3  Classes

10.4  Decorators

To make a function accessible in templates, that function must be decorated with infogami.utils.view.public decorator. Since templates can be written by untrusted users, utmost care must be taken when exposing a function in templates.

10.5  Security Considerations

The basic flow is: the user submits a url or HTML form; the FCGI handler matches the URL patch against a series of regexps to dispatch the request to the proper plugin; the plugin receives the GET or POST data and creates results as Python objects (e.g. a list of strings or ints); the plugin then loads a template (designated in the plugin code) which receives the result objects as arguments. The template code then expands the result to HTML which gets presented to the user's browser. Thus, a user can "re-skin" the application by modifying the static HTML or the script code in the templates that he or she uses. A few initial templates (such as the application's top-level HTML page) are included in the application source tree, and the rest are created and edited through the wiki interface.

Plugins are written by the application developers and are part of the source code tree for the application. Templates are entered and updated through the wiki HTTP interface, are stored in the database just like other wiki pages, and can be written by arbitary users. Thus, templates are potentially hostile and therefore require security precautions during execution. Templates give a flexible means of creating presentation HTML, but for security reasons, operations requiring access to system functions must be implemented in plugins, not templates. Templates can be application-wide or can be specific to a given user account.

  Authors

aaronsw, anand, solrize and webchick.

  References

  Template and Plugin Examples

The following examples illustrate what is possible with the Infogami template language.

Search Template
Price Check
Data Visualization

  Creating a new Infogami Type (step-by-step tutorial)

This PR details most of the code that was implemented for the scaffolding of the Tag type in 2023. The following sections provide context to the steps necessary for scaffolding a new Infogami type in the Open Library.

openlibrary/core/models.py

To begin, you need to create a placeholder class for your type and register it as a ‘Thing’ within the Infobase, the database for Infogami types. To do so, you have to create a new class for the new Infogami type within core/models.py. Within this class, you can include functionalities to interact with the new Infogami type. You then register the class as a new ‘Thing’ within the register_models function. You also have to register the new Infogami type within the Open Library using the register_types function. The following regex string '^/ indicates to web.py that URLs with the following path pattern belong to your Infogami type.

openlibrary/core/schema.py + openlibrary/core/infobase_schema.sql

Even though we have successfully registered a new ‘Thing’ within Infobase, we have not yet created the right database tables where new instances of this new type will live. If you are developing locally, you can add a new Infogami table group (using schema.add_table_group) within schema.py. (_Note: The schema.add_table_group line in schema.py was commented out in the merged PR for the creation of Tag type because it was causing errors within the production environment that we did not encounter locally)

If you plan to have unique Open Library sequences for your new type, you will also have to add a new sequence to the Infobase in schema.py using schema.add_seq('/type/. With that, every time an instance of your new Infogami type is created and added to the table group, a new sequence (e.g. ‘OL123Z’) will also be generated. This acts as a unique key to this instance, allowing us to view it in the Open Library site via this key (e.g. openlibrary.org/works/OL18020194W).

With the addition of your new table group and sequence, when you run docker compose up, you should see SQL DDL script for your new type being generated in your terminal. Look out for it and copy the chunk of script into the infobase_schema.sql file.

You will need to manually run the SQL DDL to create the new tables and sequences in your local Postgres database for Open Library. These are the commands to run the SQL DDL via terminal (assuming you already have the Open Library's docker containers running using docker compose up):

Here is an example of the SQL DDL that Infogami generated when we ran docker compose up that we then added within Postgres manually, both locally and in production:

openlibrary/plugins/openlibrary/types

You can create a new .type file in this folder for your new type. This file specifies the schema that states what fields can be used with your type and what format those fields are in. Those are used to generate view and edit templates which can then be further customized as a particular type requires. For example, this is the view page for Subject Tags and this is the YML format of its schema. Here is an example of a new Tag we created which implements the Tag schema. You can include some basic fields such as Name and Description for now as it can be amended at a later stage.

Congrats, you have now successfully scaffolded a new Infogami type. Now, to create, view and edit instances of your Infogami type via the Open Library UI, you will have to create API endpoints and HTML pages.

openlibrary/templates

Within this folder, create a new folder to store the add.html file to send http POST requests via html form to create your Infogami type. You can reference other add.html templates implemented. The name and value fields of your HTML input elements are the key-value pairs passed in your HTTP request payload and picked up by web.py for further Python manipulations.

openlibrary/templates/type

Within this folder, create a new folder to store the edit.html and view.html files to view and edit your Infogami type. When you enter the view (e.g. openlibrary.org/tags/OL1T) and edit (e.g. openlibrary.org/tags/OL1T/edit) pages of your Infogami type, web.py will automatically fetch the corresponding instance of your type via the sequence indicated in the URL path and pass its metadata to populate these HTML pages. If you are unsure what is the pattern of the path for your type, check core/models.py for the pattern you indicated in the register_types function.

openlibrary/plugins/upstream

With the HTML forms set up, you need to create a file in this folder to include the script for your API endpoints. You will need to create 2 classes, one for adding and the other for editing your Infogami type. You will need to specify GET and POST endpoint functions. As mentioned earlier, when a HTTP request is made via the HTML forms, the payload data can be accessed using web.py via the web.input function. For more details on implementing the endpoints, refer to other files in this folder.

That concludes all the technical implementations needed to create a new Infogami type.

  Code Repository

The infogami code repository can be found here.

  Installing the Software

Installation instructions can be found here.

  Troubleshooting

Q. I can't change my page type when I create a new page in the wiki using my new template set. Whenever I change the page type (to 'edition' instead of 'page', for instance), it reverts back to the original type.

A. Make sure your edit template contains the box for 'page type'. It is possible that it was removed when the form was designed. The absence of this box means that the template loses the type information whenever it is saved.

Q. I am trying to use custom graphics using the type for my particular page template set, however it generates this error: error in processing template: AttributeError: 'str' object has no attribute 'png' (falling back to default template). What gives?

A. Aha. You have to say ${key}.png because otherwise it thinks you want the png field of the key variable (as in $page.doi).

Q. Why doesn't the HTML formatting I enter in my edit template show up correctly on my view template? I added carraige returns and page breaks in all the appropriate places, but it runs together in one long paragraph.

A. You need to use the '$:format' command to support HTML formatting. For instance, to properly format the appearance of a biography on the author view page you would need to add the following the script in the appropriate place on the page:

$:format(page.bio)

Q. My HTML code for any images I add via a page edit form displays on the view page. For example, instead of a photo of Dave Eggers, I see "<img src="http://ia350629.us.archive.org/3/items/ol-images/daveeggers.gif" align="left">"

A. Again, you need to use:

$:format(page.bio) instead of $page.bio

if you want to support HTML formatting.

Q. Help! My edit template for [this page] is royally hosed. I was trying to do something fancy with [form layout | javascript | etc. ] and broke the edit template so badly that I can't get back in and correct it.

A. Use the emergency override to fall back to a very simplified template version and see if you can make your corrections there: