Onegov Core API

Framework

The Framework provides a base Morepath application that offers certain features for applications deriving from it:

  • Virtual hosting in conjunction with onegov.server.

  • Access to an SQLAlchemy session bound to a specific Postgres schema.

  • A cache backed by redis, shared by multiple processes.

  • An identity policy with basic rules, permissions and role.

  • The ability to serve static files and css/js assets.

Using the framework does not really differ from using Morepath:

from onegov.core.framework import Framework

class MyApplication(Framework):
    pass
class onegov.core.framework.Framework[source]

Baseclass for Morepath OneGov applications.

request_class

alias of onegov.core.request.CoreRequest

dsn = None

holds the database connection string, if there is a database connected

schema = None

holdes the current schema associated with the database connection, set by and derived from set_application_id().

classmethod form(*args, **kw)

framework directives

classmethod cronjob(*args, **kw)

Register a cronjob.

classmethod static_directory(*args, **kw)

Registers a static files directory.

classmethod template_variables(*args, **kw)

Registers a set of global template variables for chameleon templates.

Only exists once per application. Template variables with conflicting keys defined in child applications override the keys with the same name in the parent application. Non-conflicting keys are kept individually.

Example:

@App.template_variables()
def get_template_variables(request):
    return {
        'foo': 'bar'
    }

sets the same-site cookie directive, (may need removal inside iframes)

request_cache = None

the request cache is initialised/emptied before each request

modules

Provides access to modules used by the Framework class. Those modules cannot be included at the top because they themselves usually include the Framework.

Admittelty a bit of a code smell.

property has_database_connection

onegov.core has good integration for Postgres using SQLAlchemy, but it doesn’t require a connection.

It’s possible to have Onegov applications using a different database or not using one at all.

property has_filestorage

Returns true if fs is available.

handle_exception(exception, environ, start_response)[source]

Stops database connection errors from bubbling all the way up to our exception handling services (sentry.io).

configure_application(**cfg)[source]

Configures the application. This function calls all methods on the current class which start with configure_, passing the configuration as keyword arguments.

The core itself supports the following parameters. Additional parameters are made available by extra configure_ methods.

Dsn

The database connection to use. May be None.

See onegov.core.orm.session_manager.setup()

Base

The declarative base class used. By default, onegov.core.orm.Base is used.

Identity_secure

True if the identity cookie is only transmitted over https. Only set this to False during development!

Identity_secret

A random string used to sign the identity. By default a random string is generated. The drawback of this is the fact that users will be logged out every time the application restarts.

So provide your own if you don’t want that, but be sure to have a really long, really random key that you will never share with anyone!

Redis_url

The redis url used (default is ‘redis://localhost:6379/0’).

File_storage

The file_storage module to use. See http://docs.pyfilesystem.org/en/latest/filesystems.html

File_storage_options

A dictionary of options passed to the __init__ method of the file_storage class.

The file storage is expected to work as is. For example, if fs.osfs.OSFS is used, the root_path is expected exist.

The file storage can be shared between different onegov.core applications. Each application automatically gets its own namespace inside this space.

Always_compile_theme

If true, the theme is always compiled - no caching is employed.

Allow_shift_f5_comple

If true, the theme is recompiled if shift+f5 is done on the browser (or shift + reload button click).

Csrf_secret

A random string used to sign the csrf token. Make sure this differs from identity_secret! The algorithms behind identity_secret and the csrf protection differ. If the same secret is used we might leak information about said secret.

By default a random string is generated. The drawback of this is the fact that users won’t be able to submit their forms if the app is restarted in the background.

So provide your own, but be sure to have a really long, really random string that you will never share with anyone!

Csrf_time_limit

The csrf time limit in seconds. Basically the amount of time a user has to submit a form, from the time it’s rendered.

Defaults to 1’200s (20 minutes).

Mail

A dictionary keyed by e-mail category (i.e. ‘marketing’, ‘transactional’) with the following subkeys:

  • host: The mail server to send e-mails from.

  • port: The port used for the mail server.

  • force_tls: True if TLS should be forced.

  • username: The mail username

  • password: The mail password

  • sender: The mail sender

  • use_directory: True if a mail directory should be used

  • directory: Path to the directory that should be used

Mail_use_directory

If true, mails are stored in the maildir defined through mail_directory. There, some other process is supposed to pick up the e-mails and send them.

Mail_directory

The directory (maildir) where mails are stored if if mail_use_directory is set to True.

Sql_query_report

Prints out a report sql queries for each request, unless False. Valid values are:

  • ‘summary’ (only show the number of queries)

  • ‘redundant’ (show summary and the actual redundant queries)

  • ‘all’ (show summary and all executed queries)

Do not use in production!

Profile

If true, profiles the request and stores the result in the profiles folder with the following format: YYYY-MM-DD hh:mm:ss.profile

Do not use in production!

Print_exceptions

If true, exceptions are printed to stderr. Note that you should usually configure logging through onegov.server. This is mainly used for certain unit tests where we use WSGI more directly.

set_application_id(application_id)[source]

Set before the request is handled. Gets the schema from the application id and makes sure it exists, if a database connection is present.

get_cache(name, expiration_time)[source]

Gets a cache bound to this application id.

property session_cache

A cache that is kept for a long-ish time.

property cache

A cache that might be invalidated frequently.

property settings

Returns the settings bound to this app.

property application_id_hash

The application_id as hash, use this if the applicaiton_id can be read by the user -> this obfuscates things slightly.

object_by_path(path, with_view_name=False)[source]

Takes a path and returns the object associated with it. If a scheme or a host is passed it is ignored.

Be careful if you use this function with user provided urls, we load objects here, not views. Therefore no security restrictions apply.

The first use case of this function is to provide a generic copy/paste functionality. There, we only allow urls to be copied which have been previously signed by the server.

Safeguards like this are necessary if the user has the ability to somehow influence the path!

permission_by_view(model, view_name=None)[source]

Returns the permission required for the given model and view_name.

The model may be an instance or a class.

If the view cannot be evaluated, a KeyError is raised.

session

Alias for self.session_manager.session.

postman[source]

Returns a Mailthon postman configured with the mail settings in the settings view. See http://mailthon.readthedocs.org/ for more information.

send_marketing_email(*args, **kwargs)[source]

Sends an e-mail categorised as marketing. This includes but is not limited to:

  • Announcements

  • Newsletters

  • Promotional E-Mails

When in doubt, send a marketing e-mail. Transactional e-mails are sacred and should only be used if necessary. This ensures that the important stuff is reaching our customers!

send_transactional_email(*args, **kwargs)[source]

Sends an e-mail categorised as transactional. This is limited to:

  • Welcome emails

  • Reset passwords emails

  • Notifications

  • Weekly digests

  • Receipts and invoices

send_email(reply_to, category='marketing', receivers=(), cc=(), bcc=(), subject=None, content=None, encoding='utf-8', attachments=(), headers={}, plaintext=None)[source]

Sends a plain-text e-mail using postman to the given recipients. A reply to address is used to enable people to answer to the e-mail which is usually sent by a noreply kind of e-mail address.

E-mails sent through this method are bound to the current transaction. If that transaction is aborted or not commited, the e-mail is not sent.

Usually you’ll use this method inside a request, where transactions are automatically commited at the end.

For more complex use cases have a look at http://mailthon.readthedocs.org/.

send_zulip(subject, content)[source]

Sends a hipchat message asynchronously.

We are using the stream message method of zulip: https://zulipchat.com/api/stream-message

Returns the thread object to allow waiting by calling join.

static_files

A list of static_files paths registered through the onegov.core.directive.StaticDirectoryAction directive.

To register a static files path:

@App.static_directory()
def get_static_directory():
    return 'static'

For this to work, server_static_files has to be set to true.

When a child application registers a directory, the directory will be considered first, before falling back to the parent’s static directory.

serve_static_files

Returns True if /static files should be served. Needs to be enabled manually.

Note that even if the static files are not served, /static path is still served, it just won’t return anything but a 404.

Note also that static files are served publicly. You can override this in your application, but doing that and testing for it is on you!

See also: onegov.core.static.

application_bound_identity(userid, groupid, role)[source]

Returns a new morepath identity for the given userid, group and role, bound to this application.

property filestorage

Returns a filestorage object bound to the current application. Based on this nifty module:

http://docs.pyfilesystem.org/en/latest/

The file storage returned is guaranteed to be independent of other applications (the scope is the application_id, not just the class).

There is no guarantee as to what file storage backend is actually used. It’s quite possible that the file storage will be somewhere online in the future (e.g. S3).

Therefore, the urls for the file storage should always be acquired through onegov.core.request.CoreRequest.filestorage_link().

The backend is configured through configure_application().

For a list of methods available on the resulting object, consult this list: http://docs.pyfilesystem.org/en/latest/interface.html.

If no filestorage is available, this returns None. See self.has_filestorage.

WARNING: Files stored in the filestorage are available publicly! All someone has to do is to guess the filename. To get an unguessable filename use onegov.core.filestorage.random_filename().

The reason for this is the fact that filestorage may be something external in the future.

This should not deter you from using this for user uploads, though you should be careful. If you want to be sure that your application stores files locally, use some other ways of storing those files.

Example:

from onegov.core import filestorage

filename = filestorage.random_filename()
app.filestorage.settext(filename, 'Lorem Ipsum')

# returns either an url like '/files/4ec56cc005c594880a...'
# or maybe 'https://amazonaws.com/onegov-cloud/32746/220592/q...'
request.filestorage_link(filename)
property themestorage

Returns a storage object meant for themes, shared by all applications.

Only use this for theming, nothing else!

property theme_options

Returns the application-bound theme options.

translations

Returns all available translations keyed by langauge.

chameleon_translations

Returns all available translations for chameleon.

locales

Returns all available locales in a set.

default_locale

Returns the default locale.

property identity_secret

The identity secret, guaranteed to only be valid for the current application id.

property csrf_secret

The identity secret, guaranteed to only be valid for the current application id.

sign(text)[source]

Signs a text with the identity secret.

The text is signed together with the application id, so if one application signs a text another won’t be able to unsign it.

unsign(text)[source]

Unsigns a signed text, returning None if unsuccessful.

onegov.core.framework.get_webasset_url()[source]

The webassets url needs to be unique so we can fix it before returning the generated html. See fix_webassets_url_factory().

onegov.core.framework.get_cronjobs_enabled()[source]

If this value is set to False, all cronjobs are disabled. Only use this during testing. Cronjobs have no impact on your application, unless there are defined cronjobs, in which case they are there for a reason.

onegov.core.framework.default_content_security_policy()[source]

The default content security policy used throughout OneGov.

onegov.core.framework.default_policy_apply_factory()[source]

Adds the content security policy report settings from the yaml.

onegov.core.framework.activate_session_manager_factory(app, handler)[source]

Activate the session manager before each transaction.

onegov.core.framework.close_session_after_request_factory(app, handler)[source]

Closes the session after each request.

This frees up connections that are unused, without costing us any request performance from what I can measure.

Directive

class onegov.core.directives.HtmlHandleFormAction(model, form, render=None, template=None, load=None, permission=None, internal=False, **predicates)[source]

Register Form view.

Basically wraps the Morepath’s html directive, registering both POST and GET (if no specific request method is given) and wrapping the view handler with wrap_with_generic_form_handler().

The form is either a class or a function. If it’s a function, it is expected to return a form class when given an instance of the model.

The form may also be None, which is useful under special circumstances. Generally you don’t want that though.

Example:

@App.form(model=Root, template='form.pt',
          permission=Public, form=LoginForm)
def handle_form(self, request, form):
    if form.submitted():
        # do something if the form was submitted with valid data
    else:
        # do something if the form was not submitted or not
        # submitted correctly

    return {}  # template variables
perform(obj, *args, **kwargs)[source]

Do whatever configuration is needed for obj.

Needs to be implemented by the Action subclass.

Raise a DirectiveError to indicate that the action cannot be performed due to incorrect configuration.

Parameters
  • obj – the object that the action should be performed for. Typically a function or a class object.

  • **kw – a dictionary of configuration objects as specified by the config class attribute.

onegov.core.directives.fetch_form_class(form_class, model, request)[source]

Given the form_class defined with the form action, together with model and request, this function returns the actual class to be used.

onegov.core.directives.query_form_class(request, model, name=None)[source]

Queries the app configuration for the form class associated with the given model and name. Take this configuration for example:

@App.form(model=Model, form_class=Form, name='foobar')
...

The form class defined here can be retrieved as follows:

query_form_class(request, model=Model, name=’foobar’)

onegov.core.directives.wrap_with_generic_form_handler(obj, form_class)[source]

Wraps a view handler with generic form handling.

This includes instantiatng the form with translations/csrf protection and setting the correct action.

class onegov.core.directives.CronjobAction(hour, minute, timezone, once=False)[source]

Register a cronjob.

identifier(**kw)[source]

Returns an immutable that uniquely identifies this config.

Needs to be implemented by the Action subclass.

Used for overrides and conflict detection.

If two actions in the same group have the same identifier in the same configurable, those two actions are in conflict and a ConflictError is raised during commit().

If an action in an extending configurable has the same identifier as the configurable being extended, that action overrides the original one in the extending configurable.

Parameters

**kw – a dictionary of configuration objects as specified by the config class attribute.

Returns

an immutable value uniquely identifying this action.

perform(func, cronjob_registry)[source]

Do whatever configuration is needed for obj.

Needs to be implemented by the Action subclass.

Raise a DirectiveError to indicate that the action cannot be performed due to incorrect configuration.

Parameters
  • obj – the object that the action should be performed for. Typically a function or a class object.

  • **kw – a dictionary of configuration objects as specified by the config class attribute.

class onegov.core.directives.StaticDirectoryAction[source]

Registers a static files directory.

identifier(staticdirectory_registry)[source]

Returns an immutable that uniquely identifies this config.

Needs to be implemented by the Action subclass.

Used for overrides and conflict detection.

If two actions in the same group have the same identifier in the same configurable, those two actions are in conflict and a ConflictError is raised during commit().

If an action in an extending configurable has the same identifier as the configurable being extended, that action overrides the original one in the extending configurable.

Parameters

**kw – a dictionary of configuration objects as specified by the config class attribute.

Returns

an immutable value uniquely identifying this action.

perform(func, staticdirectory_registry)[source]

Do whatever configuration is needed for obj.

Needs to be implemented by the Action subclass.

Raise a DirectiveError to indicate that the action cannot be performed due to incorrect configuration.

Parameters
  • obj – the object that the action should be performed for. Typically a function or a class object.

  • **kw – a dictionary of configuration objects as specified by the config class attribute.

class onegov.core.directives.TemplateVariablesAction[source]

Registers a set of global template variables for chameleon templates.

Only exists once per application. Template variables with conflicting keys defined in child applications override the keys with the same name in the parent application. Non-conflicting keys are kept individually.

Example:

@App.template_variables()
def get_template_variables(request):
    return {
        'foo': 'bar'
    }
identifier(templatevariables_registry)[source]

Returns an immutable that uniquely identifies this config.

Needs to be implemented by the Action subclass.

Used for overrides and conflict detection.

If two actions in the same group have the same identifier in the same configurable, those two actions are in conflict and a ConflictError is raised during commit().

If an action in an extending configurable has the same identifier as the configurable being extended, that action overrides the original one in the extending configurable.

Parameters

**kw – a dictionary of configuration objects as specified by the config class attribute.

Returns

an immutable value uniquely identifying this action.

perform(func, templatevariables_registry)[source]

Do whatever configuration is needed for obj.

Needs to be implemented by the Action subclass.

Raise a DirectiveError to indicate that the action cannot be performed due to incorrect configuration.

Parameters
  • obj – the object that the action should be performed for. Typically a function or a class object.

  • **kw – a dictionary of configuration objects as specified by the config class attribute.

Static Files

Static files in OneGov applications are served under /static.

By default the /static folder of the application is used, relative to the path of the application class. Files in that folder are available to everyone if enabled:

from onegov.core.framework import Framework

class App(Framework):
    serve_static_files = True

By default, the /static path is registered, but returns 404s. This prevents accidental serving of static files.

To change the path to be served have a look at onegov.core.framework.Framework.static_files().

Note that this is not meant to serve css/javascript files, rather it’s a way to serve images, documents and other things that are really static.

Files served through this mechanism support the If-Modified-Since header.

If you need to serve something on another path you can:

class Favicon(StaticFile):
    pass

@App.path(model=Favicon, path='favicon.ico')
def get_favicon(app, absorb):
    return StaticFile.from_application(app, 'favicon.ico')
class onegov.core.static.StaticFile(path, version=None)[source]

Defines a static file served by the application.

classmethod from_application(app, absorb)[source]

Absorbs all /static/* paths and returns StaticFile instances with the path set to a subpath of onegov.core.Framework.static_files().

For security reasons this subpath is required to actually be inside the static_files folder. No symlinks are allowed.

onegov.core.static.view_static_file(self, request)[source]

Renders the given static file in the browser.

Filestorage

Filestorage is a way to store files locally or on a remote server, with the interface being the same, no matter where the files are stored.

Based on http://docs.pyfilesystem.org/en/latest/

See onegov.core.framework.Framework.filestorage for more information.

onegov.core.filestorage.random_filename()[source]

Returns a random filename that can’t be guessed.

class onegov.core.filestorage.FilestorageFile(path)[source]

Defines a static file served by the application. The difference between this and onegov.core.static.StaticFile is the storage.

Filestorage files are stored per application_id locally or on the cloud. Static files are the same for the whole application class and they are deployed statically. That means they are not content, but part of the distribution.

Note that this is only used if the file is local. Files stored in the filestorage should be linked using onegov.core.request.CoreRequest.filestorage_link(), which might result in a local path, for which this class is used. Or it might result in a remote path that is served by some different webserver.

onegov.core.filestorage.view_filestorage_file(self, request)[source]

Renders the given filestorage file in the browser.

onegov.core.filestorage.delete_static_file(self, request)[source]

Deletes the given filestorage file. By default the permission is Private. An application using the framework can override this though.

Since a DELETE can only be sent through AJAX it is protected by the same-origin policy. That means that we don’t need to use any CSRF protection here.

That being said, browser bugs and future changes in the HTML standard make it possible for this to happen one day. Therefore, a time-limited token must be passed as query parameter to this function.

New tokens can be acquired through request.new_csrf_token.

Theme

onegov.core provides very basic theming support with the following features:

  • Themes can be external to onegov.core

  • Themes can be used by onegov.core applications

  • Themes can be compiled and the result is shared between applications

Themes are not meant to be switched at runtime though, they are statically defined by the applications. The idea is to have a way to share themes between applications, not to have powerful skinning features where the user can change the theme at any given time.

A theme is basically a CSS file. There is no way to define images/icons and so on. The only way to do that is to include the image in the css file (which is not that crazy of an idea anyway).

To write a theme, create a class providing the properties/methods of Theme.

To use a theme you need to define the following setting:

@App.setting(section='core', name='theme')
def get_theme():
    return Theme()

To override the options passed to a theme, override the following function in your application class:

class App(Framework):

    @property
    def theme_options(self):
        return {'background': 'red'}

To include the theme in your html, call the following function in your template:

<link rel="stylesheet" type="text/css" href="${request.theme_link}">

Note that for the theme to work you need to define a filestorage. See onegov.core.framework.Framework.configure_application().

class onegov.core.theme.Theme[source]

Describres a onegov.core theme.

A onegov theme is any kind of compiled or non-compiled css file. The core expects a single css file that stays the same as long as the same options are passed to the compile function.

Some framework based themes might required the ability to serve javascript at the same time. This is not done here. Such javascript needs to be manually included through webassets.

This is due to the fact that onegov themes are not meant to be switched around. An application will chose one theme and stick with it (and test against it).

property name

The name of the theme, must be unique.

property default_options

The default options of the theme, will be overritten by options passed to compile().

compile(options={})[source]

Returns a single css that represents the theme.

onegov.core.theme.get_filename(theme, options={})[source]

Returns a unique filename for the given theme and options.

onegov.core.theme.compile(storage, theme, options={}, force=False)[source]

Generates a theme and stores it in the filestorage, returning the path to the theme.

If the theme already exists and doesn’t need recompiling, it will not compile the theme again.

Storage

The Pyfilesystem storage to store the files in.

Theme

The theme instance that should be compiled.

Options

The hashable options passed to the theme.

Force

If true, the compilation is done in any case.

onegov.core.theme.get_theme()[source]

Defines the default theme, which is no theme.

class onegov.core.theme.ThemeFile(path)[source]
onegov.core.theme.get_themestorage_file(app, absorb)[source]

Serves the theme files.

Request

class onegov.core.request.Message(text, type)
text

Alias for field number 0

type

Alias for field number 1

class onegov.core.request.ReturnToMixin[source]

Provides a safe and convenient way of using return-to links.

Return-to links are links with an added ‘return-to’ query parameter which points to the url a specific view (usually with a form) should return to, once all is said and done.

There’s no magic involved. If a view should honor the return-to paramter, it should use request.redirect instead of morepath.redirect.

If no return-to parameter was specified, rqeuest.redirect is a transparent proxy to morepath.redirect.

To create a link:

url = request.return_to(original_url, redirect)

To honor the paramter in a view, if present:

return request.redirect(default_url)

Do not use the return-to parameter directly. Redirect parameters are notorious for being used in phising attacks. By using return_to and redirect you are kept safe from these attacks as the redirect url is signed and verified.

For the same reason you should not allow the user-data for return-to links. Those are meant for internally generated links!

class onegov.core.request.CoreRequest(*args, **kwargs)[source]

Extends the default Morepath request with virtual host support and other useful methods.

Virtual hosting might be supported by Morepath directly in the future: https://github.com/morepath/morepath/issues/185

Override the link_prefix with the application base path provided by onegov.server, because the default link_prefix contains the hostname, which is not useful in our case - we’ll add the hostname ourselves later.

x_vhm_host

Return the X_VHM_HOST variable or an empty string.

X_VHM_HOST acts like a prefix to all links generated by Morepath. If this variable is not empty, it will be added in front of all generated urls.

x_vhm_root

Return the X_VHM_ROOT variable or an empty string.

X_VHM_ROOT is a bit more tricky than X_VHM_HOST. It tells Morepath where the root of the application is situated. This means that the value of X_VHM_ROOT must be an existing path inside of Morepath.

We can understand this best with an example. Let’s say you have a Morepath application that serves a blog under /blog. You now want to serve the blog under a separate domain, say blog.example.org.

If we just served Morepath under blog.example.org, we’d get urls like this one:

blog.example.org/blog/posts/2014-11-17-16:00

In effect, this subdomain would be no different from example.org (without the blog subdomain). However, we want the root of the host to point to /blog.

To do this we set X_VHM_ROOT to /blog. Morepath will then automatically return urls like this:

blog.example.org/posts/2014-11-17-16:00
url

Returns the current url, taking the virtual hosting in account.

transform(url)[source]

Applies X_VHM_HOST and X_VHM_ROOT to the given url (which is expected to not contain a host yet!).

Extends the default link generating function of Morepath.

Extends the default class link generating function of Morepath.

Takes the given filestorage path and returns an url if the path exists. The url might point to the local server or it might point to somehwere else on the web.

Returns the link to the current theme. Computed once per request.

The theme is automatically compiled and stored if it doesn’t exist yet, or if it is outdated.

browser_session

Returns a browser_session bound to the request. Works via cookies, so requests without cookies won’t be able to use the browser_session.

The browser session is bound to the application (by id), so no session data is shared between the applications.

If no data is written to the browser_session, no session_id cookie is created.

get_form(form_class, i18n_support=True, csrf_support=True, data=None, model=None)[source]

Returns an instance of the given form class, set up with the correct translator and with CSRF protection enabled (the latter doesn’t work yet).

Form classes passed to this function (or defined through the App.form directive) may define a on_request method, which is called after the request has been bound to the form and before the view function is called.

translate(text)[source]

Transalates the given text, if it’s a translatable text.

translator

Returns the translate function for basic string translations.

default_locale

Returns the default locale.

locale

Returns the current locale of this request.

html_lang

The language code for the html tag.

get_translate(for_chameleon=False)[source]

Returns the translate method to the given request, or None if no such method is availabe.

For_chameleon

True if the translate instance is used for chameleon (which is special).

message(text, type)[source]

Adds a message with the given type to the messages list. This messages list may then be displayed by an application building on onegov.core.

For example:

Four default types are defined on the request for easier use:

success() warning() info() alert()

The messages are stored with the session and to display them, the template using the messages should call consume_messages().

consume_messages()[source]

Returns the messages, removing them from the session in the process. Call only if you can be reasonably sure that the user will see the messages.

success(text)[source]

Adds a success message.

warning(text)[source]

Adds a warning message.

info(text)[source]

Adds an info message.

alert(text)[source]

Adds an alert message.

is_logged_in

Returns True if the current request is logged in at all.

agent

Returns the user agent, parsed by ua-parser.

has_permission(model, permission)[source]

Returns True if the current user has the given permission on the given model.

has_access_to_url(url)[source]

Returns true if the current user has access to the given url.

The domain part of the url is completely ignored. This method should only be used if you have no other choice. Loading the object by url first is slower than if you can get the object otherwise.

The initial use-case for this function is the to parameter in the login view. If the to-url is accessible anyway, we skip the login view.

If we can’t find a view for the url, a KeyError is thrown.

exclude_invisible(models)[source]

Excludes models invisble to the current user from the list.

is_visible(model)[source]

Returns True if the given model is visible to the current user.

In addition to the is_public check, this checks if the model is secret and should therefore not be visible (though it can still be reached via URL).

is_public(model)[source]

Returns True if the current user has the Public permission for the given model.

is_personal(model)[source]

Returns True if the current user has the Personal permission for the given model.

is_private(model)[source]

Returns True if the current user has the Private permission for the given model.

is_secret(model)[source]

Returns True if the current user has the Secret permission for the given model.

current_role

Returns the user-role of the current request, if logged in. Otherwise, None is returned.

has_role(*roles)[source]

Returns true if the current user has any of the given roles.

new_csrf_token(salt=None)[source]

Returns a new CSRF token. A CSRF token can be verified using is_valid_csrf_token().

Note that forms do their own CSRF protection. This is meant for CSRF protection outside of forms.

onegov.core uses the Synchronizer Token Pattern for CSRF protection: https://www.owasp.org/index.php/Cross-Site_Request_Forgery_%28CSRF%29_Prevention_Cheat_Sheet

New CSRF tokens are signed usign a secret attached to the session (but not sent out to the user). Clients have to return the CSRF token they are given. The token has to match the secret, which the client doesn’t know. So an attacker would have to get access to both the cookie and the html source to be able to forge a request.

Since cookies are marked as HTTP only (no javascript access), this even prevents CSRF attack combined with XSS.

assert_valid_csrf_token(signed_value=None, salt=None)[source]

Validates the given CSRF token and returns if it was created by new_csrf_token(). If there’s a mismatch, a 403 is raised.

If no signed_value is passed, it is taken from request.params.get(‘csrf-token’).

new_url_safe_token(data, salt=None)[source]

Returns a new URL safe token. A token can be deserialized using load_url_safe_token().

load_url_safe_token(data, salt=None, max_age=3600)[source]

Deserialize a token created by new_url_safe_token().

If the token is invalid, None is returned.

ORM

onegov.core.orm.orm_cached(policy)[source]

The decorator use to setup the cache descriptor.

See the onegov.core.orm.cache docs for usage.

class onegov.core.orm.SessionManager(dsn, base, engine_config=None, session_config=None, pool_config=None)[source]

Holds sessions and creates schemas before binding sessions to schemas.

Threadsafe in theory, but not tested or well thought out. No global state though, so two different session managers will manage different sessions/schemas.

register_engine(engine)[source]

Takes the given engine and registers it with the schema switching mechanism. Maybe used to register external engines with the session manager.

If used like this, make sure to call bind_session() before using the session provided by the external engine.

register_session(session)[source]

Takes the given session and registers it with zope.sqlalchemy and various orm events.

activate()[source]

Sets the currently active session_manager - this is basically a global variable we require because we hook into the orm/query events where we don’t know yet which session is going to be used and therefore cannot be sure about the used session manager either

For example, the Document model below needs access to the current session to get the current locale, but since it is instantiated without any session information, this can’t be known without a global variable:

document = Document(localized_title=”Das Dokument”)

We can only work around that with a global variable:

session_manager.activate() document = Document(localized_title=”Das Dokument”)

As a result session managers need to be activated, or they can’t be used (at least for translated columns). To do so in tests, use:

session_manager.activate()

To ease the use of testing the last session_manager is however also automatically activated when the schema is set, which covers most use-cases outside of testing

deactivate()[source]

Sets the currently active session manager to None, if it is equal to self.

set_locale(default_locale, current_locale)[source]

Sets the default locale and the current locale so it may be shared with the translated ORM columns.

Note that the locales may be NONE. It is up to the consumer of these settings to assert their existence.

dispose()[source]

Closes the connection to the server and cleans up. This only needed for testing.

is_valid_schema(schema)[source]

Returns True if the given schema looks valid enough to be created on the database with no danger of SQL injection or other unwanted sideeffects.

set_current_schema(schema)[source]

Sets the current schema in use. The current schema is then used to bind the session to a schema. Note that this can’t be done in a functional way. We need the current schema to generate a new scope.

I would very much prefer to bind this to the session but this is not at all straight-forward with SQLAlchemy.

I tried a solution like this one, but it’s not good enough, because it still relies on some kind of global stage, even if it’s set locally.

Ideally a value could be bound to the session and an event would trigger every time the connection is used with that session. We could then set the schema on the connection every time that happens.

For now, the global option is okay though, because in practice we only set the schema once per request and we don’t do threading anyway.

create_schema(schema, validate=True)[source]

Creates the given schema. If said schema exists, expect this method to throw an error. If you use set_current_schema(), this is invoked automatically if needed. So you usually shouldn’t have to care about this function.

create_schema_if_not_exists(schema, validate=True)[source]

Creates the given schema if it doesn’t exist yet.

bind_session(session)[source]

Bind the session to the current schema.

session()[source]

Returns a new session or an existing session. Sessions with different schemas are kept independent, though they might reuse each others connections.

This means that a session retrieved thusly:

mgr = SessionManager('postgres://...')
mgr.set_current_schema('foo')
session = mgr.session()

Will not see objects attached to this session:

mgr.set_current_schema('bar')
session = mgr.session()
list_schemas(namespace=None)[source]

Returns a tuple containing all schemas defined in the current database.

is_schema_found_on_database(schema)[source]

Returns True if the given schema exists on the database.

create_required_extensions()[source]

Creates the required extensions once per lifetime of the manager.

ensure_schema_exists(schema)[source]

Makes sure the schema exists on the database. If it doesn’t, it is created.

onegov.core.orm.query_schemas(connection, namespace=None)[source]

Yields all schemas or the ones with the given namespace.

onegov.core.orm.as_selectable(query, alias=None)[source]

Takes a raw SQL query and turns it into a selectable SQLAlchemy expression using annotations in comments.

Expects to find a SELECT statement like this:

SELECT
    foo,  -- Text
    bar   -- Integer
FROM
    foobar

The so created selectable can be used by SQLAlchemy:

from sqlalchemy import select

query = as_selectable('''
    SELECT
        foo, -- Text
        bar  -- Integer
    FROM
        foobar
''')

session.execute(select(query.c).where(query.c.foo == 'bar'))
onegov.core.orm.translation_hybrid = <sqlalchemy_utils.i18n.TranslationHybrid object>

A translation hybrid integrated with OneGov Core. See also: http://sqlalchemy-utils.readthedocs.org/en/latest/internationalization.html

onegov.core.orm.find_models(base, is_match)[source]

Finds the ORM models in the given ORM base class that match a filter.

The filter is called with each class in the instance and it is supposed to return True if it matches.

For example, find all SQLAlchemy models that use ContentMixin:

from onegov.core.orm.mixins import ContentMixin
find_models(base, is_match=lambda cls: issubclass(cls, ContentMixin))
class onegov.core.orm.debug.Timer[source]

A timer that works just like a stopwatch.

onegov.core.orm.debug.print_query(query)[source]

Pretty prints the given query.

onegov.core.orm.debug.analyze_sql_queries(report='summary')[source]

Analyzes the sql-queries executed during its context. There are three levels of information (report argument):

  • ‘summary’ (only show the number of queries)

  • ‘redundant’ (show summary and the actual redundant queries)

  • ‘all’ (show summary and all executed queries)

Use this with a with_statement:

with analyze_sql_queries():
    ... # <- analyzes all sql queries that happen inside here
class onegov.core.orm.abstract.adjacency_list.MoveDirection[source]

Describs the possible directions for the AdjacencyListCollection.move() method.

above = 1

Moves the subject above the target

below = 2

Moves the subject below the target

onegov.core.orm.abstract.adjacency_list.sort_siblings(siblings, key, reverse=False)[source]

Sorts the siblings by the given key, writing the order to the database.

class onegov.core.orm.abstract.adjacency_list.AdjacencyList(title, parent=None, **kwargs)[source]

An abstract AdjacencyList implementation representing a Tree.

id = Column(None, Integer(), table=None, primary_key=True, nullable=False)

the id fo the db record (only relevant internally) do not change this id after creation as that would destroy the tree

name = Column(None, Text(), table=None, nullable=False)

the name of the item - think of this as the id or better yet the url segment e.g. parent-item/child-item

automatically generated from the title if not provided

type = Column(None, Text(), table=None)

the type of the item, this can be used to create custom polymorphic subclasses of this class. See http://docs.sqlalchemy.org/en/improve_toc/ orm/extensions/declarative/inheritance.html.

order = Column(None, Integer(), table=None, default=ColumnDefault(65536))

the order of the items - items are added at the end by default

property sort_key

The sort key used for sorting the siblings if the title changes.

title = Column(None, Text(), table=None, nullable=False)

the human readable title of the item

property root

Returns the root of this item.

property ancestors

Returns all ancestors of this item.

property siblings

Returns a query that includes all siblings, including the item itself.

property path

Returns the path of this item.

absorb

Alias for path. This is a convenience feature for Morepath if a path is absorbed.

See http://morepath.readthedocs.org/en/latest/paths_and_linking.html?highlight=absorb#absorbing

class onegov.core.orm.abstract.adjacency_list.AdjacencyListCollection(session)[source]

A base class for collections working with AdjacencyList.

static sort_key(item)[source]

The sort key with which the items are sorted into their siblings.

query(ordered=True)[source]

Returns a query using AdjacencyListCollection.__listclass__.

property roots

Returns the root elements.

by_id(item_id)[source]

Takes the given page id and returns the page. Try to keep this id away from the public. It’s not a security problem if it leaks, but it’s not something the public can necessarly count on.

If possible use the path instead.

by_path(path, ensure_type=None)[source]

Takes a path and returns the page associated with it.

For example, given this hierarchy:

Page(name='documents', id=0, parent_id=None)
    Page(name='readme', id=1, parent_id=0)
    Page(name='license', id=2, parent_id=0)

The following query would return the Page with the name ‘license’:

paths.by_path('documents/license')

Slashes at the beginning or end are ignored, so the above is equal to:

paths.by_path('/documents/license')
paths.by_path('/documents/license/')
paths.by_path('documents/license/')

Lookups by path are currently quite wasteful. To get the root of a page nested deeply one has to walk the ascendants of the page one by one, triggering a number of queries.

Should this actually become a bottleneck (it might be fine), we should probably implement a materialized view that is updated whenever a page changes.

See:

<http://schinckel.net/2014/11/22/ postgres-tree-shootout-part-1%3A-introduction./>

<http://schinckel.net/2014/11/27/ postgres-tree-shootout-part-2%3A-adjacency-list-using-ctes/>

get_unique_child_name(name, parent)[source]

Takes the given name or title, normalizes it and makes sure that it’s unique among the siblings of the item.

This is achieved by adding numbers at the end if there are overlaps.

For example, root becomes root-1 if root exists. root-2 if root-1 exists and so on.

add(parent, title, name=None, type=None, **kwargs)[source]

Adds a page to the given parent.

delete(page)[source]

Deletes the given page and all it’s desecendants!.

move(subject, target, direction)[source]

Takes the given subject and moves it somehwere in relation to the target.

Subject

The item to be moved.

Target

The item above which or below which the subject is moved.

Direction

The direction relative to the target. Either MoveDirection.above if the subject should be moved above the target, or MoveDirection.below if the subject should be moved below the target.

Generic associations are an interesting SQLAlchemy design pattern that allow for collections of records to be attached to other models by mere inheritance.

Generic associations are useful in situations where a model should be attachable to a number of third-party models. For example, we might say that a comment model should be attacheable to any other model (say a page or a ticket). In such an instance generic associations offer the benefit of being simple to integrate on the third-party model.

Additionally we can define generic methods that work on all third-party models that inherit from the associated model (in our example imagine a “Commentable” class that leads to the automatic attachment of comments to Page/Ticket models).

See also: https://github.com/zzzeek/sqlalchemy/blob/master/examples/ generic_associations/table_per_association.py

Note that you do not want to use these associations in lieue of simple relationships between two models. The standard way of doing this leads to a strong relationship on the database and is easier to change and reason about.

Generic associations are meant for generic usecases. These currently include payments (any model may be payable) and files (any model may have files attached). Other generic associations should be introduced along these lines.

A single model may be associated to any number of other models. For example:

                        ┌─────────────┐
                  ┌─────│ Reservation │
┌────────────┐    │     └─────────────┘
│  Payment   │◀───┤
└────────────┘    │     ┌─────────────┐
                  └─────│    Form     │
                        └─────────────┘

Here, Payment is associable (through the Payable mixin). Reservation and Form in turn inherit from Payable.

This all is probably best understood in an example:

class Payment(Base, Associable):

__tablename__ == ‘payments’

class Payable(object):

payment = associated(Payment, ‘payment’, ‘one-to-one’)

class Product(Base, Payable):

__tablename__ == ‘products’

This results in a product model which has a payment attribute attached to it. The payments are stored in the payments table, products in the products table. The link between the two is established in the automatically created payments_for_products table.

onegov.core.orm.abstract.associable.associated(associated_cls, attribute_name, cardinality='one-to-many', uselist='auto', backref_suffix='__tablename__')[source]

Creates an associated attribute. This attribute is supposed to be defined on the mixin class that will establish the generic association if inherited by a model.

Parameters
  • associated_cls – The class which the model will be associated with.

  • attribute_name – The name of the attribute used on the mixin class.

  • cardinality – May be ‘one-to-one’, ‘one-to-many’ or ‘many-to-many’. Cascades are used automatically, unless ‘many-to-many’ is used, in which case cascades are disabled.

  • uselist – True if the attribute on the inheriting model is a list. Use ‘auto’ if this should be automatically decided depending on the cardinality.

Example:

class Adress(Base, Associable):
    pass

class Addressable(object):
    address = associated(Address, 'address', 'one-to-one')

class Company(Base, Addressable):
    pass
class onegov.core.orm.abstract.associable.Associable[source]

Mixin to enable associations on a model. Only models which are associable may be targeted by :func:`associated`_

classmethod association_base()[source]

Returns the model which directly inherits from Associable.

All associated classes are registered through this method. This yields the following benefits:

  1. We gain the ability to query all the linked records in one query. This is hard otherwise as each Payable class leads to its own association table which needs to be queried separately.

  2. We are able to reset all created backreferences. This is necessary during tests. SQLAlchemy keeps these references around and won’t let us re-register the same model multiple times (which outside of tests is completely reasonable).

Returns a query chain with all records of all models which attach to the associable model.

class onegov.core.orm.mixins.ContentMixin[source]

Mixin providing a meta/content JSON pair. Meta is a JSON column loaded with each request, content is a JSON column loaded deferred (to be shown only in the detail view).

class onegov.core.orm.mixins.dict_property(attribute, key=None, default=None)[source]

Enables access of dictionaries through properties.

Usage:

class Model(ContentMixin):
    access_times = dict_property('meta')

This creates a property that accesses the meta directory with the key ‘access_times’. The key is implicitly copied from the definition.

Another way of writing this out would be:

class Model(ContentMixin):
    access_times = dict_property('meta', 'access_times')

As is apparent, the ‘access_times’ key is duplicated in this case. Usually you do not need to provide the name. The exception being if you want the property name and the dictionary key to differ:

class Model(ContentMixin):
    access_times = dict_property('meta', 'access')

Here, the key in the dictionary is ‘access’, while the property is ‘access_times’.

Since we often use the same kind of dictionaries we can use the builtin properties that are scoped to a specific dictionary:

class Model(ContentMixin):
    access_times = meta_property()

This is equivalent to the initial example.

We can also create our own scoped properties as follows:

foo_property = dict_property_factory(‘foo’)

class Model(object):

foo = {}

bar = foo_property()

Here, model.bar would read model.foo[‘bar’].

Dict properties are compatible with typical python properties, so the usual getter/setter/deleter methods are also available:

class Model(ContentMixin):
    content = meta_property()

    @content.setter
    def set_content(self, value):
        self.meta['content'] = value
        self.meta['content_html'] = to_html(value)

    @content.deleter
    def del_content(self):
        del self.meta['content']
        del self.meta['content_html']
class onegov.core.orm.mixins.TimestampMixin[source]

Mixin providing created/modified timestamps for all records.

The columns are deferred loaded as this is primarily for logging and future forensics.

force_update()[source]

Forces the model to update by changing the modified parameter.

last_change

Returns the self.modified if not NULL, else self.created.

class onegov.core.orm.types.HSTORE(text_type=None)[source]

Extends the default HSTORE type to make it mutable by default.

class onegov.core.orm.types.JSON(*args, **kwargs)[source]

A JSONB based type that coerces None’s to empty dictionaries.

That is, this JSONB column does not have NULL values, it only has falsy values (an empty dict).

impl

alias of sqlalchemy.dialects.postgresql.json.JSONB

process_bind_param(value, dialect)[source]

Receive a bound parameter value to be converted.

Subclasses override this method to return the value that should be passed along to the underlying TypeEngine object, and from there to the DBAPI execute() method.

The operation could be anything desired to perform custom behavior, such as transforming or serializing data. This could also be used as a hook for validating logic.

This operation should be designed with the reverse operation in mind, which would be the process_result_value method of this class.

Parameters
  • value – Data to operate upon, of any type expected by this method in the subclass. Can be None.

  • dialect – the Dialect in use.

process_result_value(value, dialect)[source]

Receive a result-row column value to be converted.

Subclasses should implement this method to operate on data fetched from the database.

Subclasses override this method to return the value that should be passed back to the application, given a value that is already processed by the underlying TypeEngine object, originally from the DBAPI cursor method fetchone() or similar.

The operation could be anything desired to perform custom behavior, such as transforming or serializing data. This could also be used as a hook for validating logic.

Parameters
  • value – Data to operate upon, of any type expected by this method in the subclass. Can be None.

  • dialect – the Dialect in use.

This operation should be designed to be reversible by the “process_bind_param” method of this class.

class onegov.core.orm.types.UTCDateTime[source]

Stores dates as UTC.

Internally, they are stored as timezone naive, because Postgres takes the local timezone into account when working with timezones. Values taken and values returned are forced to be timezone-aware though.

impl

alias of sqlalchemy.sql.sqltypes.DateTime

process_bind_param(value, engine)[source]

Receive a bound parameter value to be converted.

Subclasses override this method to return the value that should be passed along to the underlying TypeEngine object, and from there to the DBAPI execute() method.

The operation could be anything desired to perform custom behavior, such as transforming or serializing data. This could also be used as a hook for validating logic.

This operation should be designed with the reverse operation in mind, which would be the process_result_value method of this class.

Parameters
  • value – Data to operate upon, of any type expected by this method in the subclass. Can be None.

  • dialect – the Dialect in use.

process_result_value(value, engine)[source]

Receive a result-row column value to be converted.

Subclasses should implement this method to operate on data fetched from the database.

Subclasses override this method to return the value that should be passed back to the application, given a value that is already processed by the underlying TypeEngine object, originally from the DBAPI cursor method fetchone() or similar.

The operation could be anything desired to perform custom behavior, such as transforming or serializing data. This could also be used as a hook for validating logic.

Parameters
  • value – Data to operate upon, of any type expected by this method in the subclass. Can be None.

  • dialect – the Dialect in use.

This operation should be designed to be reversible by the “process_bind_param” method of this class.

class onegov.core.orm.types.UUID(as_uuid=True)[source]

The UUID type used throughout OneGov. The base is always the UUID type defined by SQLAlchemy for Postgres, but we change the default to actually handle the values as uuid.UUID values.

Another approach could be the following:

https://github.com/seantis/libres/blob/master/libres/db/models/types/uuid_type.py

We can switch to this any time.

class onegov.core.orm.types.LowercaseText(*args, **kwargs)[source]

Text column that forces all text to be lowercase.

impl

alias of sqlalchemy.sql.sqltypes.TEXT

omparator_factory

alias of sqlalchemy_utils.operators.CaseInsensitiveComparator

process_bind_param(value, dialect)[source]

Receive a bound parameter value to be converted.

Subclasses override this method to return the value that should be passed along to the underlying TypeEngine object, and from there to the DBAPI execute() method.

The operation could be anything desired to perform custom behavior, such as transforming or serializing data. This could also be used as a hook for validating logic.

This operation should be designed with the reverse operation in mind, which would be the process_result_value method of this class.

Parameters
  • value – Data to operate upon, of any type expected by this method in the subclass. Can be None.

  • dialect – the Dialect in use.

Cache

Provides caching methods for onegov.core.

Onegov.core uses dogpile for caching: https://dogpilecache.readthedocs.org/

Unlike dogpile onegov.core does not provide a global region however. The cache is available through the app:

request.app.cache.set('key', 'value')

Global caches in a multi-tennant application are a security vulnerability waiting to be discovered. Therefore we do not do that!

This means that this won’t be available:

from x import cache
@cache.cache_on_arguments()
def my_function():
    return 'foobar'

Syntactic sugar like this will be provided through decorators inside this module in the future. For example, we could write one that is usable on all morepath views:

@App.view(...)
@cache.view()
def my_view():
    return '<html>...'

But no such method exists yet.

Currently there is one cache per app that never expires (though values will eventually be discarded by redis if the cache is full).

class onegov.core.cache.DillSerialized(*args, **kwargs)[source]

A proxy backend that pickles all non-byte values using dill.

get(key)[source]

Retrieve a value from the cache.

The returned value should be an instance of CachedValue, or NO_VALUE if not present.

set(key, value)[source]

Set a value in the cache.

The key will be whatever was passed to the registry, processed by the “key mangling” function, if any. The value will always be an instance of CachedValue.

get_multi(keys)[source]

Retrieve multiple values from the cache.

The returned value should be a list, corresponding to the list of keys given.

New in version 0.5.0.

set_multi(mapping)[source]

Set multiple values in the cache.

mapping is a dict in which the key will be whatever was passed to the registry, processed by the “key mangling” function, if any. The value will always be an instance of CachedValue.

When implementing a new CacheBackend or cutomizing via ProxyBackend, be aware that when this method is invoked by Region.get_or_create_multi(), the mapping values are the same ones returned to the upstream caller. If the subclass alters the values in any way, it must not do so ‘in-place’ on the mapping dict – that will have the undesirable effect of modifying the returned values as well.

New in version 0.5.0.

Browser Session

class onegov.core.browser_session.BrowserSession(cache, token, on_dirty=<function BrowserSession.<lambda>>)[source]

A session bound to a token (session_id cookie). Used to store data about a client on the server.

Instances should be called browser_session to make sure we don’t confuse this with the orm sessions.

Used by onegov.core.request.CoreRequest.

Example:

browser_session = request.browser_session browser_session.name = ‘Foo’ assert client.session.name == ‘Foo’ del client.session.name

This class can act like an object, through attribute access, or like a dict, through square brackets. Whatever you prefer.

pop(name, default=None)[source]

Returns the value for the given name, removing the value in the process.

Internationalization (i18n)

Provides tools and methods for internationalization (i18n).

Applications wishing to use i18n need to define two settings:

i18n.localedirs

A list of gettext locale directories. The first directory is considered to be the main directory, all other directories are added to the translation object as fallbacks.

i18n.default_locale

The fallback locale that is used if no locale more suitable to the user could be found.

For example:

from onegov.core.framework import Framework
from onegov.core import utils

class App(Framework):
    pass

@TownApp.setting(section='i18n', name='localedirs')
def get_i18n_localedirs():
    return [
        utils.module_path('onegov.town', 'locale')
        utils.module_path('onegov.form', 'locale')
    ]

@TownApp.setting(section='i18n', name='default_locale')
def get_i18n_default_locale():
    return 'en'
onegov.core.i18n.get_i18n_localedirs()[source]

Returns the gettext locale dir.

onegov.core.i18n.get_i18n_locales()[source]

Returns the the locales actually used.

onegov.core.i18n.get_i18n_default_locale()[source]

Returns the fallback language to use if the user gives no indication towards his preference (throught the request or anything).

onegov.core.i18n.get_i18n_locale_negotiatior()[source]

Returns the language negotiator, which is a function that takes the current request as well as a list of available languages and returns the angauge that should be used based on that information.

onegov.core.i18n.default_locale_negotiator(locales, request)[source]

The default locale negotiator.

Will select one of the given locales by:

  1. Examining the ‘locale’ cookie which will be preferred if the language in the cookie actually exists

  2. Selecting the best match from the accept_language header

If no match can be found, None is returned. It is the job of caller to deal with that (possibly getting get_i18n_default_locale()).

onegov.core.i18n.pofiles(localedir)[source]

Takes the given directory and yields the language and the path of all pofiles found under */LC_MESSAGES/*.po.

onegov.core.i18n.get_translations(localedirs)[source]

Takes the given gettext locale directories and loads the po files found. The first found po file is assumed to be the main translations file (and should for performance reasons contain most of the translations). The other po files are added as fallbacks.

The pofiles are compiled on-the-fly, using polib. This makes mofiles unnecessary.

Returns a dictionary with the keys being languages and the values being GNUTranslations instances.

onegov.core.i18n.wrap_translations_for_chameleon(translations)[source]

Takes the given translations and wraps them for use with Chameleon.

onegov.core.i18n.translation_chain(translation)[source]

Yields the translation chain with a generator.

onegov.core.i18n.get_translation_bound_meta(meta_class, translations)[source]

Takes a wtforms Meta class and combines our translate class with the one provided by WTForms itself.

onegov.core.i18n.get_translation_bound_form(form_class, translate)[source]

Returns a form setup with the given translate function.

onegov.core.i18n.merge(translations)[source]

Takes the given translations (a list) and merges them into each other. The translations at the end of the list are overwritten by the translations at the start of the list.

This is preferrable to adding fallbacks, as they increases the average complexity of the lookup function each time.

Note that the translations are not copied before merging, so that means all existing translations are changed during this processed. To avoid this, clone the translation first (see clone()).

Returns

The last GNUTranslations object with all other translation objects merged into it. The first element overrides the second and so on.

onegov.core.i18n.clone(translation)[source]

Clones the given translation, creating an independent copy.

class onegov.core.i18n.SiteLocale(locale, to)[source]

A model representing the locale of the site.

Use this model to enable the user to change his locale through a path.

Templating (Chameleon)

Integrates the Chameleon template language.

This is basically a copy of more.chameleon, with the additional inclusion of a gettext translation function defined by onegov.core.i18n.

To use a chameleon template, applications have to specify the templates directiory, in addition to inheriting from onegov.core.framework.Framework.

For example:

from onegov.core.framework import Framework

class App(Framework):
    pass

@App.template_directory()
def get_template_directory():
    return 'templates'

@App.path()
class Root(object):
    pass

@App.html(model=Root, template='index.pt')
def view_root(self, request):
    return {
        'title': 'The Title'
    }

The folder can either be a directory relative to the app class or an absolute path.

onegov.core.templates.decode_string()

Decode the bytes using the codec registered for encoding.

encoding

The encoding with which to decode the bytes.

errors

The error handling scheme to use for the handling of decoding errors. The default is ‘strict’ meaning that decoding errors raise a UnicodeDecodeError. Other possible values are ‘ignore’ and ‘replace’ as well as any other name registered with codecs.register_error that can handle UnicodeDecodeErrors.

class onegov.core.templates.TemplateLoader(*args, **kwargs)[source]

Extends the default page template loader with the ability to lookup macros in various folders.

class onegov.core.templates.MacrosLookup(search_paths, name='macros.pt')[source]

Takes a list of search paths and provides a lookup for macros.

This means that when a macro is access through this lookup, it will travel up the search path of the template loader, to look for the macro and return with the first match.

As a result, it is possible to have a macros.pt file in each search path and have them act as if they were one file, with macros further up the list of paths having precedence over the macros further down the path.

For example, given the search paths ‘foo’ and ‘bar’, foo/macros.pt could define ‘users’ and ‘page’, while bar/macros.pt could define ‘users’ and ‘site’. In the lookup this would result in ‘users’ and ‘page’ being loaded loaded from foo and ‘site’ being loaded from bar.

onegov.core.templates.get_template_loader(template_directories, settings)[source]

Returns the Chameleon template loader for templates with the extension .pt.

onegov.core.templates.get_chameleon_render(loader, name, original_render)[source]

Returns the Chameleon template renderer for the required template.

onegov.core.templates.render_template(template, request, content, suppress_global_variables='infer')[source]

Renders the given template. Use this if you need to get the rendered value directly. If oyu render a view, this is not needed!

By default, mail templates (templates strting with ‘mail’) skip the inclusion of global variables defined through the template_variables directive.

onegov.core.templates.render_macro(macro, request, content, suppress_global_variables=True)[source]

Renders a chameleon.zpt.template.Macro like this:

layout.render_macro(layout.macros['my_macro'], **vars)

This code is basically a stripped down version of this: https://github.com/malthe/chameleon/blob/257c9192fea4b158215ecc4f84e1249d4b088753/src/chameleon/zpt/template.py#L206.

As such it doesn’t treat chameleon like a black box and it will probably fail one day in the future, if Chameleon is refactored. Our tests will detect that though.

Cronjobs

Provides a way to specify cronjobs which should be called at an exact time.

Example:

# A job that runs at 08:00 exactly
@App.cronjob(hour=8, minute=0, timezone='Europe/Zurich')
def cleanup_stuff(request):
    pass

# A job that runs every 15 minutes
@App.cronjob(hour='*', minute='*/15', timezone='UTC')
def do_interesting_things(request):
    pass

Functions registered like this will be called through a regular request. That means they are just like any other view and have all the power associated with it.

Warnings

The function is called once for each application id! So when the time comes for your function to be called, you can expect many calls on a busy site. Each application id also gets its own thread, which is not terribly efficient. It is expected that this behaviour will change in the future with an improved scheduling mechanism.

Also note that the scheduler will offset your function automatically by up to 30 seconds, to mitigate against the trampling herd problem.

As a result you want to use this feature sparingly. If possible do clean up on user action (either manually, or when the user changes something somewhat related). For example, we clean up old reservation records whenever a new reservation is received.

This also means that it is better to have a cronjob that runs once a day on a specific time (say at 13:37 each day), than a cronjob that runs exactly on 00:00 when a lot of other things might be scheduled.

Also the more a cronjob is called the costlier it is for the site (a job that runs every minute is naturally more heavy on resources than one run every 15 minutes).

Finally note that cronjobs for any given application id are only run once the first request to the application with that id has been made. The reason for this is the fact that the background thread needs a real request to be initialized.

In other words, if nobody visits the website the cronjob runs on, then the cronjobs won’t run.

onegov.core.cronjobs.parse_cron(value, type)[source]

Minimal cron style interval parser. Currently only supports this:

  • -> Run every hour, minute

*/2 -> Run every other hour, minute 2 -> Run on the given hour, minute

Returns a tuple, iterator or generator.

class onegov.core.cronjobs.Job(function, hour, minute, timezone, once=False, url=None)[source]

A single cron job.

runtimes(today)[source]

Generates the runtimes of this job on the given day, excluding runtimes in the past.

next_runtime(today=None)[source]

Returns the time (epoch) when this job should be run next, not taking into account scheduling concerns (like when it has run last).

If no runtime is found for today, a runtime is searched tomorrow. If that doesn’t work, no next runtime exists. This would be an error, since we do not currently support things like weekly cronjobs.

property id

Internal id signed by the application. Used to access the job through an url which must be unguessable, but the same over many processes.

The signature is only possible if self.app is present, which can only be set after the instance has already been created.

See as_request_call().

as_request_call(request)[source]

Returns a new job which does the same as the old job, but it does so by calling an url which will execute the original job.

class onegov.core.cronjobs.ApplicationBoundCronjobs(request, jobs)[source]

A daemon thread which runs the cronjobs it is given in the background.

The cronjobs are not actually run in the same thread. Instead a request to the actual cronjob is made. This way the thread is pretty much limited to basic IO work (GET request) and the actual cronjob is called with a normal request.

Basically there is no difference between calling the url of the cronjob at a scheduled time and using this cronjob.

This also avoids any and all locking problems as we don’t have to write OneGov applications in a thread safe way. The actual work is always done on the main thread.

Each application id is meant to have its own thread. To avoid extra work we make sure that only one thread per application exists on each server. This is accomplished by holding a local lock during the lifetime of the thread.

WARNING This doesn’t work on distributed systems. That is if the onegov processes are distributed over many servers the lock isn’t shared.

This can be achieved, but it’s difficult to get right, so for now we do not implement it, as we do not have distributed systems yet.

run()[source]

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

onegov.core.cronjobs.get_job(app, id)[source]

The internal path to the cronjob. The id can’t be guessed.

onegov.core.cronjobs.run_job(self, request)[source]

Executes the job.

Layout

class onegov.core.layout.Layout(model, request)[source]

Contains useful methods related to rendering pages in html. Think of it as an API that you can rely on in your templates.

The idea is to provide basic layout functions here, if they are usful for any kind of html application. You should then extend the core layout classes with your own.

timezone = <DstTzInfo 'Europe/Zurich' LMT+0:34:00 STD>

The timezone is currently fixed to ‘Europe/Zurich’ since all our business with onegov is done here. Once the need arises, we should lookup the timezone from the IP of the user, or use a javascript library that sets the timezone for the user session.

There’s also going to be the case where we want the timezone set specifically for a certain layout (say a reservation of a room, where the room’s timezone is relevant). This is why this setting should remain close to the layout, and not necessarily close to the request.

time_format = 'HH:mm'

http://cldr.unicode.org/translation/date-time-patterns

Classes inheriting from Layout may add their own formats, as long as they end in _format. For example:

class MyLayout(Layout):
    my_format = 'dd.MMMM'
    my_skeleton_format = 'skeleton:yMMM'

MyLayout().format_date(dt, 'my')

XXX this is not yet i18n and could be done better

app

Returns the application behind the request.

chunks(*args, **kwargs)[source]

See onegov.core.utils.chunks().

csrf_token

Returns a csrf token for use with DELETE links (forms do their own thing automatically).

csrf_protected_url(url)[source]

Adds a csrf token to the given url.

format_date(dt, format)[source]

Takes a datetime and formats it according to local timezone and the given format.

isodate(date)[source]

Returns the given date in the ISO 8601 format.

parse_isodate(string)[source]

Returns the given ISO 8601 string as datetime.

number_symbols[source]

Returns the locale specific number symbols.

format_number(number, decimal_places=None, padding='')[source]

Takes the given numer and formats it according to locale.

If the number is an integer, the default decimal places are 0, otherwise 2.

property view_name

Returns the view name of the current view, or None if it is the default view.

Note: This relies on morepath internals and is experimental in nature!

class onegov.core.layout.ChameleonLayout(model, request)[source]

Extends the base layout class with methods related to chameleon template rendering.

This class assumes the existance of two templates:

  • layout.pt -> Contains the page skeleton with headers, body and so on.

  • macros.pt -> Contains chameleon macros.

template_loader

Returns the chameleon template loader.

base

Returns the layout, which defines the base layout of all town pages. See templates/layout.pt.

macros

Returns the macros, which offer often used html constructs. See templates/macros.pt.

elements

The templates used by the elements. Overwrite this with your own templates/elements.pt if neccessary.

Crypto

onegov.core.crypto.hash_password(password)[source]

The default password hashing algorithm used by onegov.

Over time the underlying algorithm may change, at which point verify_password() must issue a deprecation warning when using the old algorithm.

Note that no salt is being passed, because the algorithm we use now (bcrypt), as well as the algorithm that we might use in the future (scrypt), generate their own salt automatically by default.

The salt is then stored in the resulting hash. That means that we do not pass or store a salt ourselves.

onegov.core.crypto.verify_password(password, hash)[source]

Compares a password to a hash and returns true if they match according to the hashing algorithm used.

onegov.core.crypto.random_password(length=16)[source]

Returns a random password using the markov chain below.

onegov.core.crypto.random_token(nbytes=512)[source]

Generates an unguessable token. Generates a random string with the given number of bytes (may not be lower than 512) and hashes the result to get a token with a consistent length of 64.

Why hashing?

We could of course just create a random token with a length of 64, but that would leak the random numbers we actually create. This can be a bit of a problem if the random generator you use turns out to have some vulnerability. By hashing a larger number we hide the result of our random generator.

Doesn’t generating a hash from a larger number limit the number of tokens?

Yes it does. The number of different tokens is 2^256 after hashing, which is a number larger than all the atoms on earth (approx. 2^166). So there is a chance of a collision occuring, but it is very unlikely to ever happen.

More information:

http://wyattbaldwin.com/2014/01/09/generating-random-tokens-in-python

http://www.2uo.de/myths-about-urandom/

http://crypto.stackexchange.com/q/1401

onegov.core.crypto.stored_random_token(namespace, name)[source]

A random token that is only created once per boot of the host (assuming the host deletes all files in the /tmp folder).

This method should only be used for development and is not meant for general use!

Security

OneGov uses a very simple permissions model by default. There are no read/write permissions, just intents. That means a permission shows the intended audience.

This is the default however, any application building on top of onegov.core may of course introduce its own byzantine permission system.

class onegov.core.security.permissions.Public[source]

The general public is allowed to do this.

class onegov.core.security.permissions.Private[source]

Trusted people are allowed to do this.

class onegov.core.security.permissions.Personal[source]

Registered members are allowed to do this.

class onegov.core.security.permissions.Secret[source]

Only Demi-Gods are allowed to do this.

onegov.core.security.roles.get_roles_setting()[source]

Returns the default roles available to onegov.core applications.

Applications building on onegov.core may add more roles and permissions, or replace the existing ones entirely, though it’s not something that one should do carelessly.

The default roles are:

admin

Has access to everything

editor

Has access to most things

member

Has access their own data. Be careful though, core doesn’t know about personal data, so this is just a role to implement registered users. As with all permissions, making sure the right information is shown is up to the applications.

anonymous

Has access to public things

onegov.core.security.rules.has_permission_not_logged_in(app, identity, model, permission)[source]

This catch-all rule returns the default permission rule. It says that the permission must be part of the anonymous rule.

Models with an ‘access’ property set to ‘secret’ are prohibited from being viewed by anonymous users.

onegov.core.security.rules.has_permission_logged_in(app, identity, model, permission)[source]

This permission rule matches all logged in identities. It requires the identity to have a ‘role’ attribute. Said role attribute is used to determine if the given permission is part of the given role.

onegov.core.security.rules.may_view_http_errors_not_logged_in(app, identity, model, permission)[source]

HTTP errors may be viewed by anyone, regardeless of settings.

This is important, otherwise the HTTPForbidden/HTTPNotFound views will lead to an exception if the user does not have the Public permission.

onegov.core.security.rules.may_view_cronjobs_not_logged_in(app, identity, model, permission)[source]

Cronjobs are run anonymously from a thread and need to be excluded from the permission rules as a result.

Collection

class onegov.core.collection.SearcheableCollection(session)[source]

Requires a self.locale and self.term

static match_term(column, language, term)[source]

Usage: model.filter(match_term(model.col, ‘german’, ‘my search term’))

static term_to_tsquery_string(term)[source]

Returns the current search term transformed to use within Postgres to_tsquery function. Removes all unwanted characters, replaces prefix matching, joins word together using FOLLOWED BY.

filter_text_by_locale(column, term, locale=None)[source]

Returns an SqlAlchemy filter statement based on the search term. If no locale is provided, it will use english as language.

to_tsquery creates a tsquery value from term, which must consist of single tokens separated by the Boolean operators & (AND), | (OR) and ! (NOT).

to_tsvector parses a textual document into tokens, reduces the tokens to lexemes, and returns a tsvector which lists the lexemes together with their positions in the document. The document is processed according to the specified or default text search configuration.

property term_filter_cols

Returns a dict of column names to search in with term. Must be attributes of self.model_class.

class onegov.core.collection.Pagination[source]

Provides collections with pagination, if they implement a few documented properties and methods.

See onegov.ticket.TicketCollection for an example.

subset()[source]

Returns an SQLAlchemy query containing all records that should be considered for pagination.

property page_index

Returns the current page index (starting at 0).

page_by_index(index)[source]

Returns the page at the given index. A page here means an instance of the class inheriting from the Pagination base class.

transform_batch_query(query)[source]

Allows subclasses to transform the given query before it is used to retrieve the batch. This is a good place to add additional loading that should only apply to the batch (say joining other values to the batch which are then not loaded by the whole query).

subset_count

Returns the total number of elements this pagination represents.

batch

Returns the elements on the current page.

property offset

Returns the offset applied to the current subset.

property pages_count

Returns the number of pages.

property pages

Yields all page objects of this Pagination.

property previous

Returns the previous page or None.

property next

Returns the next page or None.

class onegov.core.collection.RangedPagination[source]

Provides a pagination that supports loading multiple pages at once.

This is useful in a context where a single button is used to ‘load more’ results one by one. In this case we need an URL that represents what’s happening on the screen (multiple pages are shown at the same time).

subset()[source]

Returns an SQLAlchemy query containing all records that should be considered for pagination.

property page_range

Returns the current page range (starting at (0, 0)).

by_page_range(page_range)[source]

Returns an instance of the collection limited to the given page range.

limit_range(page_range, direction)[source]

Limits the range to the range limit in the given direction.

For example, 0-99 will be limited to 89-99 with a limit of 10 and ‘up’. With ‘down’ it will be limited to 0-9.

transform_batch_query(query)[source]

Allows subclasses to transform the given query before it is used to retrieve the batch. This is a good place to add additional loading that should only apply to the batch (say joining other values to the batch which are then not loaded by the whole query).

subset_count

Returns the total number of elements this pagination represents.

batch

Returns the elements on the current page range.

property pages_count

Returns the number of pages.

property previous

Returns the previous page or None.

property next

Returns the next page range or None.

Cli Commands

Core Commands

Provides a framework for cli commands run against one ore more onegov cloud applications.

OneGov cli commands are usually ran against a onegov.yml config file, which may contain definitions for multiple applications. It may define multiple application with different applicaiton classes and it may contain wildcard applications which run the same application class, but contain multiple tennants for each application.

To have a command run against one or many applications we use a selector to help select the applications we want to target in a command.

In addition to selectors, onegov core cli commands provide a simple way to write a function that takes a request and an application. This function is then called for each application matching the selector, with a proper request and application context already setup (with the same characteristics as if called through an url in the browser).

Selector

A selector has the form <namespace>/<id>.

That is, it consists of the namespace of the application and it’s id.

For example:

  • /foo/bar

  • /onegov_election_day/gr

  • /onegov_town/govikon

To select non-wildcard applications we can just omit the id:

  • /foo

  • /onegov_onboarding

Finally, to select multiple applications we can use wildcards:

  • /foo/*

  • /onegov_election_day/*

  • /*/g??

Execution

To run a supported command we provide a selector as an option:

bin/onegov-core --select '/foo/*' subcommand

To find out what kind of selectors are available, we can simply run:

bin/onegov-core

Which will print out a list of selector suggestions.

Registering a Selector Based Command

To write a selector based command we first create a command group:

from onegov.core.cli import command_group
cli = command_group()

Using that command group, we can register our own commands:

@cli.command()
def my_click_command():
    pass

This command works like any other click command:

import click

@cli.command()
@click.option('--option')
def my_click_command(option):
    pass

Each command has the ability to influence the way selectors work. For example, a command which creates the path that matches the selector we can use:

@cli.command(context_settings={'creates_path': True})

By default we expect that a selector is passed. For commands which usually run against all applications we can provide a default selector:

@cli.command(context_settings={'default_selector': '*'})

Using the app/request context

For a lot of commands the easiest approach is to have a function which is called for each application with a request. This allows us to write commands which behave like they were written in a view.

To do that we register a command which returns a function with the following signature:

def handle_command(request, app):
    pass

For example:

@cli.command()
def my_click_command():

    def handle_command(request, app):
        pass

    return handle_command

Setup like this, handle_command will be called with a request for each application (and tennant). This function acts exactly like a view. Most importantly, it does not require transaction commits, because like with ordinary requests, the transaction is automatically committed if no error occurs.

Using the app configurations directly

Sometimes we don’t want to use the request/app context, or maybe we want to setup something before receiving a request.

To do this, we use the pass_group_context decorator.

For example:

from onegov.core.cli import pass_group_context

@cli.command()
@pass_group_context
def my_click_command(group_context):

    for appcfg in group_context.appcfgs:
        # do something

This is independent of the app/request context. If we return a function, the function is going to be called with the request and the app. If we do not, the command ends as expected.

Returning multiple functions

When a cli command returns multiple functions, they are run in succession.

The signature is taken into account. If there’s a ‘request’ parameter in the function, the usual request context is set up.

If there is no ‘request’ parameter in the function, it is called once per appcfg, together with the group context:

@cli.command()
def my_special_command():

    def handle_command(request, app):
        pass

    def handle_raw(group_context, appcfg):
        pass

    return (handle_command, handle_raw)

Limiting Selectors to a Single Instance

Sometimes we want to write commands which only run against a single application. A good example is a command which returns 1/0 depending on the existence of something in an application.

To do that, we use:

@cli.command(context_settings={'singular': True})
def my_click_command():
    pass

If a selector is passed which matches more than one application, the command is not executed.

onegov.core.cli.core.CONTEXT_SPECIFIC_SETTINGS = ('default_selector', 'creates_path', 'singular', 'matches_required')

GroupContext settings which may be overriden by commands

class onegov.core.cli.core.GroupContextGuard[source]

Contains methods which abort the commandline program if any condition is not met.

Used as a mixin in GroupContext.

class onegov.core.cli.core.GroupContext(selector, config, default_selector=None, creates_path=False, singular=False, matches_required=True)[source]

Provides access to application configs for group commands.

Parameters
  • selector

    Selects the applications which should be captured by a command_group().

    See Core Commands for more documentation about selectors.

  • config – The targeted onegov.yml file or an equivalent dictionary.

  • default_selector – The selector used if none is provided. If not given, a selector has to be provided.

  • creates_path

    True if the given selector doesn’t exist yet, but will be created. Commands which use this setting are expected to take a single path (no wildcards) and to create it during their runtime.

    Implies singular and matches_required.

  • singular – True if the selector may not match multiple applications.

  • matches_required – True if the selector must match at least one application.

available_schemas(appcfg)[source]

Returns all available schemas, if the application is database bound.

match_to_path(match)[source]

Takes the given match and returns the application path used in http requests.

match_to_appcfg(match)[source]

Takes the given match and returns the maching appcfg object.

property appcfgs

Returns the matching appconfigs.

Since there’s only one appconfig per namespace, we ignore the path part of the selector and only focus on the namespace:

/namespace/application_id
property available_selectors

Generates a list of available selectors.

The list doesn’t technically exhaust all options, but it returns all selectors targeting a single application as well as all selectors targeting a namespace by wildcard.

property all_wildcard_selectors

Returns all selectors targeting a namespace by wildcard.

property all_specific_selectors

Returns all selectors targeting an application directly.

property matches

Returns the specific selectors matching the context selector.

That is, a combination of namespace / application id is returned. Since we only know an exhaustive list of application id’s if we have a database connection this is currently limited to applications with one. Since we do not have any others yet that’s fine.

However if we implement a database-less application in the future which takes wildcard ids, we need some way to enumerate those ids.

See https://github.com/OneGov/onegov.core/issues/13

onegov.core.cli.core.get_context_specific_settings(context)[source]

Takes the given click context and extracts all context specific settings from it.

onegov.core.cli.core.pass_group_context(f)

Decorator to acquire the group context on a command:

@cli.command()
@pass_group_context()
def my_command(group_context):

pass

onegov.core.cli.core.command_group()[source]

Generates a click command group for individual modules.

Each individual module may have its own command group from which to run commands to. Read http://click.pocoo.org/6/commands/ to learn more about command groups.

The returned command group will provide the individual commands with an optional list of applications to operate on and it allows commands to return a callback function which will be invoked with the app config (if available), an application instance and a request.

That is to say, the command group automates setting up a proper request context.

onegov.core.cli.core.abort(msg)[source]

Prints the given error message and aborts the program with a return code of 1.

onegov.core.cli.commands.cli = <click.core.Group object>

onegov.core’s own command group

Upgrade

class onegov.core.upgrade.UpgradeState(**kwargs)[source]

Keeps the state of all upgrade steps over all modules.

onegov.core.upgrade.get_distributions_with_entry_map(key)[source]

Iterates through all distributions with entry_maps and yields each distribution along side the entry map with the given key.

onegov.core.upgrade.get_upgrade_modules()[source]

Returns all modules that registered themselves for onegov.core upgrades like this:

entry_points={
    'onegov': [
        'upgrade = onegov.mypackage.upgrade'
    ]
}

To add multiple upgrades in a single setup.py file, the following syntax may be used. This will become the default in the future:

entry_points= {
‘onegov_upgrades’: [

‘onegov.mypackage = onegov.mypackage.upgrade’

]

}

Note that the part before the ‘=’-sign should be kept the same, even if the location changes. Otherwise completed updates may be run again!

class onegov.core.upgrade.upgrade_task(name, always_run=False, requires=None, raw=False)[source]

Marks the decorated function as an upgrade task. Upgrade tasks should be defined outside classes (except for testing) - that is in the root of the module (directly in onegov/form/upgrades.py for example).

Each upgrade task has the following properties:

Name

The name of the task in readable English. Do not change the name of the task once you pushed your changes. This name is used as an id!

Always_run

By default, tasks are run once on if they haven’t been run yet, or if the module was not yet under upgrade control (it just added itself through the entry_points mechanism).

There are times where this is not wanted.

If you enable upgrades in your module and you plan to run an upgrade task in the same release, then you need to ‘always_run’ and detect yourself if you need to do any changes.

If you need to enable upgrades and run a task at the same time which can only be run once and is impossible to make idempotent, then you need to make two releases. One that enables upgade control and a second one that

Also, if you want to run an upgrade task every time you upgrade (for example, for maintenance), then you should also use it.

The easiest way to use upgrade control is to enable it from the first release on.

Another way to tackle this is to write update methods idempotent as often as possible and to always run them.

Requires

A reference to another projects upgrade task that should be run first.

For example:

requires='onegov.form:Add new form field'

If the reference cannot be found, the upgrade fails.

Raw

A raw task is run without setting up a request. Instead a database connection and a list of targeted schemas is passed. It’s up to the upgrade function to make sure the right schemas are targeted through that connection!

This is useful for upgrades which need to execute some raw SQL that cannot be executed in a request context. For example, this is a way to migrate the upgrade states table of the upgrade itself.

Raw tasks are always run and there state is not recorded. So the raw task needs to be idempotent and return False if there was no change.

Only use this if you have no other way. This is a very blunt instrument only used under special circumstances.

Note, raw tasks may be normal function OR generators. If they are generators, then with each yield the transaction is either commited or rolled-back depending on the dry-run cli argument.

Always run tasks may return False if they wish to be considered as not run (and therefore not shown in the upgrade runner).

onegov.core.upgrade.is_task(function)[source]

Returns True if the given function is an uprade task.

onegov.core.upgrade.get_module_tasks(module)[source]

Goes through a module or class and returns all upgrade tasks.

onegov.core.upgrade.get_tasks_by_id(upgrade_modules=None)[source]

Takes a list of upgrade modules or classes and returns the tasks keyed by id.

onegov.core.upgrade.get_module_order_key(tasks)[source]

Returns a sort order key which orders task_ids in order of their module dependencies. That is a task from onegov.core is sorted before a task in onegov.user, because onegov.user depends on onegov.core.

This is used to order unrelated tasks in a sane way.

onegov.core.upgrade.get_tasks(upgrade_modules=None)[source]

Takes a list of upgrade modules or classes and returns the tasks that should be run in the order they should be run.

The order takes dependencies into account.

onegov.core.upgrade.register_modules(session, modules, tasks)[source]

Sets up the state tracking for all modules. Initially, all tasks are marekd as executed, because we assume tasks to upgrade older deployments to a new deployment.

If this is a new deployment we do not need to execute these tasks.

Tasks where this is not desired should be marked as ‘always_run’. They will then manage their own state (i.e. check if they need to run or not).

This function is idempotent.

onegov.core.upgrade.register_all_modules_and_tasks(session)[source]

Registers all the modules and all the tasks.

class onegov.core.upgrade.UpgradeTransaction(context)[source]

Holds the session and the alembic operations connection together and commits/aborts both at the same time.

Not really a two-phase commit solution. That could be accomplished but for now this suffices.

class onegov.core.upgrade.UpgradeContext(request)[source]

Holdes the context of the upgrade. An instance of this is passed to each upgrade task.

class onegov.core.upgrade.RawUpgradeRunner(tasks, commit=True, on_task_success=None, on_task_fail=None)[source]

Runs the given raw tasks.

class onegov.core.upgrade.UpgradeRunner(modules, tasks, commit=True, on_task_success=None, on_task_fail=None)[source]

Runs the given basic tasks.

Utils

onegov.core.utils.local_lock(namespace, key)[source]

Locks the given namespace/key combination on the current system, automatically freeing it after the with statement has been completed or once the process is killed.

Usage:

with lock('namespace', 'key'):
    pass
onegov.core.utils.normalize_for_url(text)[source]

Takes the given text and makes it fit to be used for an url.

That means replacing spaces and other unwanted characters with ‘-‘, lowercasing everything and turning unicode characters into their closest ascii equivalent using Unidecode.

See https://pypi.python.org/pypi/Unidecode

onegov.core.utils.increment_name(name)[source]

Takes the given name and adds a numbered suffix beginning at 1.

For example:

foo => foo-1
foo-1 => foo-2
onegov.core.utils.lchop(text, beginning)[source]

Removes the beginning from the text if the text starts with it.

onegov.core.utils.rchop(text, end)[source]

Removes the end from the text if the text ends with it.

onegov.core.utils.remove_repeated_spaces(text)[source]

Removes repeated spaces in the text (‘a b’ -> ‘a b’).

onegov.core.utils.profile(filename)[source]

Profiles the wrapped code and stores the result in the profiles folder with the given filename.

onegov.core.utils.timing(name=None)[source]

Runs the wrapped code and prints the time in ms it took to run it. The name is printed in front of the time, if given.

onegov.core.utils.module_path(module, subpath)[source]

Returns a subdirectory in the given python module.

Mod

A python module (actual module or string)

Subpath

Subpath below that python module. Leading slashes (‘/’) are ignored.

onegov.core.utils.touch(file_path)[source]

Touches the file on the given path.

class onegov.core.utils.Bunch(**kwargs)[source]

A simple but handy “collector of a bunch of named stuff” class.

See http://code.activestate.com/recipes/52308-the-simple-but-handy-collector-of-a-bunch-of-named/.

For example:

point = Bunch(x=1, y=2)
assert point.x == 1
assert point.y == 2

point.z = 3
assert point.z == 3
onegov.core.utils.render_file(file_path, request)[source]

Takes the given file_path (content) and renders it to the browser. The file must exist on the local system and be readable by the current process.

onegov.core.utils.hash_dictionary(dictionary)[source]

Computes a sha256 hash for the given dictionary. The dictionary is expected to only contain values that can be serialized by json.

That includes int, decimal, string, boolean.

Note that this function is not meant to be used for hashing secrets. Do not include data in this dictionary that is secret!

onegov.core.utils.groupbylist(*args, **kwargs)[source]

Works just like Python’s itertools.groupby function, but instead of returning generators, it returns lists.

onegov.core.utils.linkify_phone(text)[source]

Takes a string and replaces valid phone numbers with html links. If a phone number is matched, it will be replaced by the result of a callback function, that does further checks on the regex match. If these checks do not pass, the matched number will remain unchanged.

onegov.core.utils.linkify(text, escape=True)[source]

Takes plain text and injects html links for urls and email addresses.

By default the text is html escaped before it is linkified. This accounts for the fact that we usually use this for text blocks that we mean to extend with email addresses and urls.

If html is already possible, why linkify it?

Note: We need to clean the html after we’ve created it (linkify parses escaped html and turns it into real html). As a consequence it is possible to have html urls in the text that won’t be escaped.

onegov.core.utils.remove_duplicate_whitespace(text)[source]

Removes whitespace that is duplicated.

For example: ‘foo bar’ becomes ‘foo bar’.

onegov.core.utils.paragraphify(text)[source]

Takes a text with newlines groups them into paragraphs according to the following rules:

If there’s a single newline between two lines, a <br> will replace that newline.

If there are multiple newlines between two lines, each line will become a paragraph and the extra newlines are discarded.

onegov.core.utils.ensure_scheme(url, default='http')[source]

Makes sure that the given url has a scheme in front, if none was provided.

onegov.core.utils.is_uuid(value)[source]

Returns true if the given value is a uuid. The value may be a string or of type UUID. If it’s a string, the uuid is checked with a regex.

onegov.core.utils.is_non_string_iterable(obj)[source]

Returns true if the given obj is an iterable, but not a string.

onegov.core.utils.pairwise(iterable)[source]

s -> (s0,s1), (s1,s2), (s2, s3), …

onegov.core.utils.chunks(iterable, n, fillvalue=None)[source]

Iterates through an iterable, returning chunks with the given size.

For example:

chunks('ABCDEFG', 3, 'x') --> [
    ('A', 'B', 'C'),
    ('D', 'E', 'F'),
    ('G', 'x', 'x')
]
onegov.core.utils.relative_url(absolute_url)[source]

Removes everything in front of the path, including scheme, host, username, password and port.

onegov.core.utils.is_subpath(directory, path)[source]

Returns true if the given path is inside the given directory.

onegov.core.utils.is_sorted(iterable, key=<function <lambda>>, reverse=False)[source]

Returns True if the iterable is sorted.

onegov.core.utils.morepath_modules(cls)[source]

Returns all morepath modules which should be scanned for the given morepath application class.

We can’t reliably know the actual morepath modules that need to be scanned, which is why we assume that each module has one namespace (like ‘more.transaction’ or ‘onegov.core’).

onegov.core.utils.scan_morepath_modules(cls)[source]

Tries to scann all the morepath modules required for the given application class. This is not guaranteed to stay reliable as there is no sure way to discover all modules required by the application class.

onegov.core.utils.get_unique_hstore_keys(session, column)[source]

Returns a set of keys found in an hstore column over all records of its table.

onegov.core.utils.makeopendir(fs, directory)[source]

Creates and opens the given directory in the given PyFilesystem.

onegov.core.utils.append_query_param(url, key, value)[source]

Appends a single query parameter to an url. This is faster than using Purl, if and only if we only add one query param.

Also this function assumes that the value is already url encoded.

class onegov.core.utils.PostThread(url, data, headers, timeout=30)[source]

POSTs the given data with the headers to the URL.

Example:

data = {'a': 1, 'b': 2}
data = json.dumps(data).encode('utf-8')
PostThread(
    'https://example.com/post',
    data,
    (
        ('Content-Type', 'application/json; charset=utf-8'),
        ('Content-Length', len(data))
    )
).start()

This only works for external URLs! If posting to server itself is needed, use a process instead of the thread!

run()[source]

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

onegov.core.utils.toggle(collection, item)[source]

Toggles an item in a set.

onegov.core.utils.binary_to_dictionary(binary, filename=None)[source]

Takes raw binary filedata and stores it in a dictionary together with metadata information.

The data is compressed before it is stored int he dictionary. Use dictionary_to_binary() to get the original binary data back.

onegov.core.utils.dictionary_to_binary(dictionary)[source]

Takes a dictionary created by binary_to_dictionary() and returns the original binary data.

onegov.core.utils.safe_format(format, dictionary, types={<class 'float'>, <class 'int'>, <class 'str'>}, adapt=None, raise_on_missing=False)[source]

Takes a user-supplied string with format blocks and returns a string where those blocks are replaced by values in a dictionary.

For example:

>>> safe_format('[user] has logged in', {'user': 'admin'})
'admin has logged in'
Parameters
  • format – The format to use. Square brackets denote dictionary keys. To literally print square bracktes, mask them by doubling (‘[[‘ -> ‘[‘)

  • dictionary – The dictionary holding the variables to use. If the key is not found in the dictionary, the bracket is replaced with an empty string.

  • types

    A set of types supported by the dictionary. Limiting this to safe types like builtins (str, int, float) ensure that no values are accidentally leaked through faulty __str__ representations.

    Note that inheritance is ignored. Supported types need to be whitelisted explicitly.

  • adapt – An optional callable that receives the key before it is used. Returns the same key or an altered version.

  • raise_on_missing – True if missing keys should result in a runtime error (defaults to False).

This is strictly meant for formats provided by users. Python’s string formatting options are clearly superior to this, however it is less secure!

onegov.core.utils.safe_format_keys(format, adapt=None)[source]

Takes a safe_format() string and returns the found keys.

onegov.core.utils.is_valid_yubikey(client_id, secret_key, expected_yubikey_id, yubikey)[source]

Asks the yubico validation servers if the given yubikey OTP is valid.

Client_id

The yubico API client id.

Secret_key

The yubico API secret key.

Expected_yubikey_id

The expected yubikey id. The yubikey id is defined as the first twelve characters of any yubikey value. Each user should have a yubikey associated with it’s account. If the yubikey value comes from a different key, the key is invalid.

Yubikey

The actual yubikey value that should be verified.

Returns

True if yubico confirmed the validity of the key.

onegov.core.utils.is_valid_yubikey_format(otp)[source]

Returns True if the given OTP has the correct format. Does not actually contact Yubico, so this function may return true, for some invalid keys.

onegov.core.utils.yubikey_otp_to_serial(otp)[source]

Takes a Yubikey OTP and calculates the serial number of the key.

The serial key is printed on the yubikey, in decimal and as a QR code.

Example:

>>> yubikey_otp_to_serial(
    'ccccccdefghdefghdefghdefghdefghdefghdefghklv')
2311522

Adapted from Java:

https://github.com/Yubico/yubikey-salesforce-client/blob/ e38e46ee90296a852374a8b744555e99d16b6ca7/src/classes/Modhex.cls

If the key cannot be calculated, None is returned. This can happen if they key is malformed.

onegov.core.utils.yubikey_public_id(otp)[source]

Returns the yubikey identity given a token.

onegov.core.utils.dict_path(dictionary, path)[source]

Gets the value of the given dictionary at the given path. For example:

>>> data = {'foo': {'bar': True}}
>>> dict_path(data, 'foo.bar')
True
onegov.core.utils.safe_move(src, dst)[source]

Rename a file from src to dst.

  • Moves must be atomic. shutil.move() is not atomic.

  • Moves must work across filesystems. Often temp directories and the cache directories live on different filesystems. os.rename() can throw errors if run across filesystems.

So we try os.rename(), but if we detect a cross-filesystem copy, we switch to shutil.move() with some wrappers to make it atomic.

Via https://alexwlchan.net/2019/03/atomic-cross-filesystem-moves-in-python

Webasset Filters

Extra webasset filters.

class onegov.core.filters.JsxFilter(**kwargs)[source]
setup()[source]

Overwrite this to have the filter do initial setup work, like determining whether required modules are available etc.

Since this will only be called when the user actually attempts to use the filter, you can raise an error here if dependencies are not matched.

Note: In most cases, it should be enough to simply define the options attribute. If you override this method and want to use options as well, don’t forget to call super().

Note: This may be called multiple times if one filter instance is used with different asset environment instances.

class onegov.core.filters.DataUriFilter(**kwargs)[source]

Overrides the default datauri filter to work around this issue:

https://github.com/miracle2k/webassets/issues/387

class onegov.core.filters.RCSSMinFilter(**kwargs)[source]

Adds the rcssmin filter (not yet included in webassets)

setup()[source]

Overwrite this to have the filter do initial setup work, like determining whether required modules are available etc.

Since this will only be called when the user actually attempts to use the filter, you can raise an error here if dependencies are not matched.

Note: In most cases, it should be enough to simply define the options attribute. If you override this method and want to use options as well, don’t forget to call super().

Note: This may be called multiple times if one filter instance is used with different asset environment instances.

Mail

onegov.core.mail.email(sender=None, receivers=(), cc=(), bcc=(), subject=None, content=None, encoding='utf8', attachments=(), category='marketing', plaintext=None)[source]

Creates an Envelope object with a HTML content, as well as a plaintext alternative generated from the HTML content.

Parameters
  • content – HTML content.

  • encoding – Encoding of the email.

  • attachments – Either a list of mailthon.enclosure.Enclosure or a list of filenames to attach to the email.

Note: this is basically a copy of mailthon.api.email(), though it adds the plaintext alternative.

class onegov.core.mail.Envelope(headers, enclosure, mail_from=None)[source]

Changes the mailthon envelope to use mime-multipart/alternative.

mime()[source]

Returns a mime object. Internally this generates a MIMEMultipart object, attaches the enclosures, then prepares it using the internal headers object.

class onegov.core.mail.MaildirTransport(host, port, maildir)[source]

A transport that pretends to be like python’s smtplib.SMTP transport, but actually just stores files into a maildir.

class onegov.core.mail.MaildirPostman(host, port, middlewares=(), options=None)[source]

A mailthon postman that stores mails to a maildir.

transport

alias of MaildirTransport

class onegov.core.datamanager.MailDataManager(postman, envelope)[source]

Takes a postman and an envelope and sends it when the transaction is commited.

Since we can’t really know if a mail can be sent until it happens, we simply log an exception if the sending failed.

class onegov.core.datamanager.FileDataManager(data, path)[source]

Writes a file when the transaction is commited.

CSV

Offers tools to deal with csv (and xls, xlsx) files.

class onegov.core.csv.CSVFile(csvfile, expected_headers=None, dialect=None, encoding=None, rename_duplicate_column_names=False, rowtype=None)[source]

Provides access to a csv file.

Parameters
  • csvfile

    The csv file to be accessed. Must be an open file (not a poth), opened in binary mode. For example:

    with open(path, 'rb') as f:
        csv = CSVFile(f)
    

  • expected_headers

    The expected headers if known. Expected headers are headers which must exist in the CSV file. There may be additional headers.

    If the headers are slightly misspelled, a matching algorithm tries to guess the correct header, without accidentally matching the wrong headers.

    See match_headers() for more information.

    If the no expected_headers are passed, no checks are done, but the headers are still available. Headers matching is useful if a user provides the CSV and it might be wrong.

    If it is impossible for misspellings to occurr, the expected headers don’t have to be specified.

  • dialect – The CSV dialect to expect. By default, the dialect will be guessed using Python’s heuristic.

  • encoding – The CSV encoding to expect. By default, the encoding will be guessed and will either be UTF-8 or CP1252.

  • rename_duplicate_column_names – It is possible to rename duplicate column names to deal with super wacky files. If this option is set and a duplicate header is found, a suffix is appended to the column name rather than throwing a DuplicateColumnNamesError.

  • rowtype

    An alternative rowtype for the resulting rows. This should be a callable that receives a rownumber key/value and all the other keys/values found in the csv. The keys are normalized and are valid Python identifiers usable as attribute names.

    Defaults to a namedtuple created using the found headers.

Once the csv file is open, the records can be acceessed as follows:

with open(path, 'rb') as f:
    csv = CSVFile(f)

    for line in csv.lines:
        csv.my_field  # access the column with the 'my_field' header
onegov.core.csv.detect_encoding(csvfile)[source]

Since encoding detection is hard to get right (and work correctly every time), we limit ourselves here to UTF-8 or CP1252, whichever works first. CP1252 is basically the csv format you get if you use windows and excel and it is a superset of ISO-8859-1/LATIN1.

onegov.core.csv.sniff_dialect(csv)[source]

Takes the given csv string and returns the dialect or raises an error. Works just like Python’s built in sniffer, just that it is a bit more conservative and doesn’t just accept any kind of character as csv delimiter.

onegov.core.csv.normalize_header(header)[source]

Normalizes a header value to be as uniform as possible.

This includes:
  • stripping the whitespace around it

  • lowercasing everything

  • transliterating unicode (e.g. ‘ä’ becomes ‘a’)

  • removing duplicate whitespace inside it

onegov.core.csv.convert_xls_to_csv(xls, sheet_name=None)[source]

Takes an XLS/XLSX file and returns a csv file using the given worksheet name or the first worksheet found.

onegov.core.csv.get_keys_from_list_of_dicts(rows, key=None, reverse=False)[source]

Returns all keys of a list of dicts in an ordered tuple.

If the list of dicts is irregular, the keys found in later rows are added at the end of the list.

Note that the order of keys is otherwise defined by the order of the keys of the dictionaries. So if ordered dictionaries are used, the order is defined. If regular dictionaries are used, the order is undefined.

Alternatively, a key and a reverse flag may be provided which will be used to order the fields. If the list of fields is specified, the key and the reverse flag is ignored.

onegov.core.csv.convert_list_of_dicts_to_csv(rows, fields=None, key=None, reverse=False)[source]

Takes a list of dictionaries and returns a csv.

If no fields are provided, all fields are included in the order of the keys of the first dict. With regular dictionaries this is random. Use an ordered dict or provide a list of fields to have a fixed order.

Alternatively, a key and a reverse flag may be provided which will be used to order the fields. If the list of fields is specified, the key and the reverse flag is ignored.

The function returns a string created in memory. Therefore this function is limited to small-ish datasets.

onegov.core.csv.convert_list_of_dicts_to_xlsx(rows, fields=None, key=None, reverse=False)[source]

Takes a list of dictionaries and returns a xlsx.

This behaves the same way as convert_list_of_dicts_to_csv().

onegov.core.csv.parse_header(csv, dialect=None, rename_duplicate_column_names=False)[source]

Takes the first line of the given csv string and returns the headers.

Headers are normalized (stripped and normalized) and expected to be unique. The dialect is sniffed, if not provided.

Returns

A list of headers in the order of appearance.

onegov.core.csv.match_headers(headers, expected)[source]

Takes a list of normalized headers and matches them up against a list of expected headers.

The headers may differ from the expected headers. This function tries to match them up using the Levenshtein distance. It does so somewhat carefully by calculating a sane distance using the input.

The passed headers are expected to be have been normalized by normalize_header(), since we usually will pass the result of parse_header() to this function.

For example:

match_headers(
    headers=['firstname', 'lastname'],
    expected=['last_name', 'first_name']
)

Results in:

['first_name', 'last_name']

If no match is possible, an MissingColumnsError, or AmbiguousColumnsError or DuplicateColumnNamesError error is raised.

Returns

The matched headers in the order of appearance.