Skip to content.

plope

Personal tools
You are here: Home » Members » chrism's Home » Proposal for Zope Scheduling Services
 
 

Proposal for Zope Scheduling Services

A proposal for low-level Zope scheduling services that I will (hopefully) code up over the next couple of days.

Motive

System- and user-level tasks need to happen on a recurring basis in Zope, such as database packing, session data garbage collection, and so on. There is no generic facility in Zope which allows us to to perform these tasks.

Scope

This proposal specifically does not deal with UI and policy issues related to automated task scheduling; it only proposes the implementation of the generic mechanisms which all automated task schedulers could make use, such as a clock and a load indicator.

Desired Features of A Generic Scheduling System

  1. Ideally, automated scheduled tasks should be able to choose to defer execution until the system is not otherwise "busy".
  2. The scheduling system should guarantee that scheduled activities will have an opportunity to run at least every N seconds.

Definitions

Clock Service: this is a simple bit of code which contacts listeners every N seconds. There is one and only one clock service per Zope instance.

Clock Listener: this is an object which listens to announcements from the clock service.

Load Indicator: a metric which allows us to determine how "busy" Zope is at a particular time.

Background: Zope Mainloop Behavior

Typically, all activity in Zope takes place based on diversions from a mainloop. The mainloop (which is based on Python's asyncore) continually performs a select on a set of file descriptors, each of which is related to one of Zope's various servers, aka "dispatchers".

If select determines that there is any activity on any file descriptor in the set it was provided, it returns the filedescriptors on which it detected the avtivity. The mainloop then performs actions against the dispatchers related to the "active" filedescriptors, reading or writing from them as necessary. Select "times out" if there is no read/write activity within a certain amount of time (typically 30 seconds). When select times out, it returns an empty set of "active" filedescriptors.

Implementation of the Load Indicator

We can get a rough sense of how "busy" Zope is at any particular time by keeping a running average of the times between mainloop select returns. Each time select returns in the mainloop, compute the difference between the current time and the last time there was a select return. Over time, average the difference. A large number indicates a not-so-busy system, a small number indicates a busy system. If the select timeout value is used as a baseline, we could compute some other scaled number, perhaps a float between 0 and 1 where 0 indicates a completely unused system and 1 indicates a completely saturated system.

This code could be placed in the Lifetime.py module and an API could be provided in this module as a function which returns the Zope instance's load value.

Implementation of the Clock Service

The clock service is very simple. It has two duties:

  • Allow callback registrations and registration deletions.
  • Call each of the registered callbacks every N seconds where N is configurable on a global basis.

This behavior could either be implemented in a separate thread that is started when Zope starts or could be caused to happen in the mainloop.

Implementation of A Clock Listener

A clock listener is any piece of code which registers a callback with the clock service. For instance, this could be a generic recurring scheduling service, like Xron, or a system-specific scheduled task like garbage collection. Its minimal duty is to register a callback with the clock service.

Issues

A clock event is just a special type of event. It is tempting to include a generic event system to support this single use case, although not necessary.

Created by chrism
Last modified 2003-12-17 10:18 PM

cool stuff chris

it occurs to me that maybe the load indicator should be a pluggable service.

It also occurs to me that I personally am never likely to plug it ;-)

Yes please!

I suspect I am not the only person who has tried Xron, encountered various problems, then rolled his own scheduler with cron and wget.

Event Service

I would be inclined to punt on a generic event service in the core for now. However I have coded a simple one for pypes which could serve as a basis (or just rip it off entirely it's ZPL'd and depends only on ZODB). It is a simple dispatcher where listeners register for "messages" by type. Messages are arbitrary objects, in fact any Python object will do. Listeners simply register that they are interested in a type of message. Listeners receive messages that are objects of that type (or a subtype) and are passed the message object.

See: http://cvs.zope.org/Packages/pypes/pypes/

1078732546

I would love to help in anyway I could on this project for the Zope2Sprint... This is needed so badly and would be an excellent learning experience...

New Event Service Product

I haven't looked at the code yet, but this was just released, and wondering what you think of it, Chris:

http://dev.legco.biz/products/ZopeScheduler