Skip to content.

plope

Personal tools
You are here: Home » Members » chrism's Home » My ZCA Thoughts Summary
 
 

My ZCA Thoughts Summary

Lately there's been some conversation on zope-dev about the future of the "zope.component" package, which is the package that provides "the Zope Component Architecture", more or less. I make some proposals here and take some positions.

The Zope community currently asserts that there is a benefit to using the current set of conventions and APIs that forms the Zope Component Architecture as a mechanism: use of its mechanisms and concepts can produce a system of higher quality and one which is more understandable than a system built without these mechanisms and concepts.

The promised benefits of obeying the conventions and constraints of the ZCA are:

  • A system based on use of the component architecture machinery is "pluggable", meaning that implementations can be swapped out for one another as necessary without changing consumer code.
  • If you've paid enough attention to detail over time that the contract spelled by an interface still matches the concrete implementations of the interface, the interface forms a natural set of obviously-correct documentation for that component. For a large system where there is an existing body of interface documentation, it is on average easier for someone new to the system to quickly get productive.
  • The concept of adaptation, where one object can be converted to another "type" through introspection of its interfaces, allows the average developer to understand the interaction between discrete components in a large system more easily than in a comparable system which doesn't use adaptation.

I hope to explain via this blog entry that the first promise is completely true, the second is at least half-true, and the third one is just plain false.

The constraints and conventions required by the ZCA are these:

  • "Interfaces" are type markers that have metadata.
  • An object may declare that it "implements" one or more interfaces.
  • An interface may optionally specify a contract: objects which implement the interface are presumed to offer this contract.
  • "Adaptation" is the act of converting an object that implements one interface to an object that implements another interface.

Let's compare the promises of following these constraints and conventions to their actual benefits.

Components as Plugpoints

Use of a ZCA "component" as a plugpoint works extremely well. These uses include: testing dependency injection, view overrides, and other extremely high-level policy plugpoints. The ZCA really shines here, as long as you use it in small doses and take care to document the resulting system narratively in terms that don't necessarily require an understanding of component architecture concepts.

I am obviously biased, but I believe repoze.bfg is such a system. However, it actually does not use any documented API defined within in the zope.component package itself. It just uses a ZCA registry as machinery and disuses much of the worldview surrounding the traditional definition of adaptation as documented in various tutorials about the ZCA. It doesn't try to use interfaces as documentation, either. The API docs are generated from the implementation classes rather than from interfaces. The presence of interfaces at all is largely just an implementation detail.

Interfaces As Documentation and Component Creation

Interfaces can act as great documentation. If you're conscientious about maintaining the interface definition for some implementation (or set of implementations), there is incontrovertible benefit to interface documentation.

However, designing a good interface is hard. Can anyone really be expected to sit down and design a formal interface that represents the contract for some component before he even codes up the first implementation of that component and some tests for it? No, of course not. Just can't happen, sorry, unless you've done this particular thing ten times before. API design is a highly iterative process: you throw ten revisions of an API away before you get one that is mostly right.

Therefore, you almost never want to promise that a particular component even has any interface until you've got that interface right in the context of some larger system full of consumers. Until you've got the interface right, the interaction between code and components that might implement some notional interface implied by a component is just an implementation detail. It might be an important implementation detail, but it's often OK if it stays one forever. Some components don't need their interfaces spelled out because they are purely an implementation detail. If there's only one or two consumers of a particular "component", it's probably OK for that component to not have a formal interface. A system that has fewer, more meaningful and highly formalized contracts is usually easier to understand than a system that has many small contracts that all have a high probability of changing radically over time, because the overhead of maintaining the interface definitions for smaller contracts makes them less likely to be true. Wrong definitions are worse than none.

So here we have this system, the ZCA, that uses interfaces as a primary mechanism to perform higher-level operations such as adaptation. You are required to declare an interface for some component to use those higher-level operations. This is required by the worldview that states that adaptation is the conversion of an object implementing one interface to another, period, full stop. Worse, once you define an interface, you are expected to maintain its definition over the lifetime of the project; ideally it should never fall out of sync with the true interface expected by the consumers of an object which implements it. This is usually unreasonable in systems with dozens or hundreds of interface definitions. On projects with many contributors over time, and many interfaces, it's not easy to ensure that interface definitions are both correct and comprehensive. You can define an interface as a plain "marker" without any attribute or method definitions. This essentially means that you promise no contract of an adaptation to this interface. But this is frowned upon in a worldview where the point of adaptation is to convert an object implementing one interface to an object implementing another. It's considered bad form, an "abuse", a cast to a void pointer.

Meanwhile, the adaptation machinery provided by the ZCA is useful in many contexts: particularly for dependency injection in unit tests, so it's extremely tempting to want to make use of it for this purpose alone. Almost all Zope-related projects use the ZCA for unit testing dependency injection. Defining "interfaces" for all your "components" is required there because it's the only way to get what you want for testing dependency injection purposes.

There should actually be very few true plugpoints in any system. Far, far, far fewer, by maybe even an order of magnitude, than those implied by use of adaptation in the typical Zope-related project. Probably nine out of ten uses of adaptation in ZCA-using application source code is there only to provide a place to hang some testing dependency injection point. These are extremely hard to tell apart from the "real" plugpoints in any given system, because they quack exactly like the plugpoint duck. Another questionable use for an interface include using one to look up a "utility" (especially an "unnamed" utility) where the utility is something that has an "obvious" API such as a dictionary. Documenting the API of this obvious thing in the interface definition is just silly. Interfaces tend to be added to any system of consequence for similarly questionable reasons.

So now at this point, if you've been following along, we've built a system that probably has incomplete documentation for its "components", because, really, there aren't any yet. Or maybe very few "true" components. We're not really smart enough to define any "components" yet. We may never be smart enough to define them properly. Our project just uses adaptation for several discrete purposes, none of them actually related to increasing comprehensibility or providing plugpoints: we've just hacked in some dependency injection points for unit tests that use the adapter machinery to good purpose. But we didn't do this to increase comprehensibility; we did it because it makes unit testing easier. We've thus effectively misdirected the casual code reader into believing that there are lots and lots of "components" with well-thought-out "interfaces" when really all we've been trying to do is to test the system more easily. The system is still in major flux, and its "interfaces" only represent some half-baked transitional state of affairs, probably not even correct. The contracts aren't truly formalized yet at all. But it looks pretty impressive.

A developer new to the project needs to be able to detect and discount the adaptations in the code done purely for unit testing dependency injection and convenience and the adaptations that imply true component plugpoints. That task is understandably difficult because we're using the same machinery for entirely different purposes.

As a result, we've actively subverted the original goal: to make the system more understandable.

Use of Adaptation Makes A System More Understandable

The idea that mere use of adaptation as a mechanism to improve large system comprehensibility for a new team member is utterly ludicrous. Mere use of the adaptation pattern adds no additional comprehensibility whatsoever to any system, small or large. Seeing "getMultiAdapter((foo, bar), IFrobnozz)" in code lends no particular insights as to the intent of the developer who wrote it. I'm not talking about syntax here either; the adaptation could be spelled "config['frobnozz](foo, bar)" and it would have the same level of comprehensibility in isolation.

Believing that mere use of any particular adaptation syntax in the code itself helps a new developer understand the intent of the adaptation is a complete fantasy. The only way to explain the intent of any particularly important adaptation is via hand-crafted narrative documentation explaining why some particular adaption takes place in a particular spot. The component architecture doesn't help at all there; you're on your own.

Writing software is hard. Writing software documentation is even harder. But it needs to be done. No tool is going to remove this requirement. Mere use of adaptation in code is no substitute whatsoever for writing high-level design documentation that explains how the system components work together. Using adaptation without explaining the "why" just hurts comprehensibility. Folks who pretend that they've made their system comprehensible simply by using adaptation are fooling themselves in the same way that folks who write doctests but don't test all their code and don't write narrative docs have fooled themselves into thinking they've written both good tests and good docs.

Worse, when we do use adaptation without explanation, a new developer must first go understand the machinery that implements adaptation to get a sense of the intent. That machinery is not particularly well-documented. There is no explanation of the lookup ordering that takes place when components are adapted at all, in particular.

As a result, we've actively subverted the original stated goal: to make the system more understandable.

Conclusions

I believe that following the current trajectory of "the ZCA" is unwise. I am happy to continue using the "zope.component" package, because its machinery works really, really well and it is useful in many scenarios, particularly for unit testing dependency injection and for building systems that can be considered "pluggable" after the fact (systems with additional documentation that explains the how and why of each true plugpoint). However, I no longer buy in to the idea that extremely casual use of the concept of adaptation defined formally as converting one interface to another laid on top of this machinery is of any benefit whatsoever when producing a large system in isolation.

I think we can improve this state of affairs via the following:

  • Create a package with only machinery which can be used to implement an adaptation pattern in the package. Document the shit out of it, like it was for your grandma. Don't impose any particular "worldview" on consumers of this machinery: in particular, do not require the use of Zope interfaces as inputs. Maybe provide extra behavior (such as inheritance) when the inputs are interfaces, but allow for registration and lookup markers that are not interfaces.
  • Base the API of what is currently zope.component on the machinery in the package I've suggested above. zope.component would be a consumer of this package, and would be free to impose the "an adaptation is the conversion of one interface to another" worldview on its consumers.
  • Provide some tools that make it easier to do unit testing dependency injection without needing to declare formal interfaces (this might be the package in the first bullet point). Or at least think up a convention where it's possible for a casual source reader to immediately know that some adaptation is present only for test dependency purposes, as opposed to a true plugpoint.

I'd be willing to work on projects that go in these directions. I'm not willing to work on projects that take the current zope.component and add APIs to it without factoring out the underlying machinery bits, though.

Created by chrism
Last modified 2009-12-03 03:26 AM

Reply

Hi Chris,

As always, there's a lot of valuable insight and reality checks here. I don't disagree with you, though I think your version of history and assumptions about the use of the ZCA (by which I mean zope.component + zope.interface) is perhaps a little different to mine.

First of all, I find the use of interface for documentation and design quite useful. I normally start by defining interfaces. They may evolve over time, but I consider interfaces.py an important contract in my packages, and a useful place to start thinking about how I'd like my code to be used by others. Python doesn't have many formalisms for segregating the "public" part of your package from the implementation details.

Equally, when reading other Zope packages, I normally start by reading *their* interfaces (at least after the readme/pypi page). Largely, I find the interfaces in the various Zope packages useful, though there are exceptions.

That said, you are absolutely right that this is not an excuse to avoid documentation. I try very hard to make sure that people reading my package's README files (normally on PyPI) have a good idea about what the package is about and how it is used.

Secondly, on the user of the adapter pattern: I don't think it's useful (or used) only for unit testing dependency injection. At least in content-management scenarios (e.g. Plone), I find that we often have a need to model different aspects of a particular object. In Zope 2, that was done with a lot of mixin classes. This didn't scale well, and led to code that was difficult to understand (have you ever dir()'d an Archetypes content object?). I often find the (big-a) Adapter pattern a useful approach for building services that are re-usable across various types of content, and yet allow customisation for individual special cases. In fact, the adapter specialisation/discrimination machinery is really powerful, and I'm somewhat surprised that you didn't mention it. I find the adapter registry provided by zope.component a useful and elegant way to implement this. It's not a panacea, and in smaller apps (like some GAE work I've done) it'd be overkill. But I haven't seen any alternatives that are better for the types of things we do in Plone.

It's a similar story with utilities. For implementing singletons, externalising policy decisions (strategy pattern, sorta), or implementing registries of homogenous objects, the utility registry is useful. It helps unify implementations of these patterns, and makes certain decisions a no-brainer where people would otherwise be making up their own (and probably inconsistent) registry implementations. This type of consistency has a benefit to consumers of the applications that use the ZCA, at least once those consumers have learned the base ZCA concepts.

I largely think your suggestions for the future direction is sensible, but I also don't know that you've done the ZCA full justice.

Martin

hi martin

I personally find reading interfaces useful in the same way I find any API documentation useful: once you know the what and why, it's extremely useful as long as it's up to date. But it's not really the first thing I ever want to read. I want to read some high-level docs that would make it clear why I should read the API documentation at all. And reading API documentation should be an *exceptional* event while I'm using the system in anger; I don't want to need to read it in order to attempt to divine some higher-level meaning from my presumption of interactions between the components documented. If a README serves that purpose, that's fine, but often a README is not enough, especially for large systems. And as you can imagine by my mentioning of "dozens or hundreds of interfaces", a lot of my questioning of the patterns in use is in the context of building large systems with the ZCA.

I don't disagree that a mechanism to do "component" lookups based on markers attached to content is useful. I actually did mention this in the first topic above; BFG uses the adapter registry for good effect. However, I don't think that this machinery should always require use of interface objects, especially if you're already documenting your object APIs some other way. I believe there are times when the addition of yet another interface makes no sense whatsoever, even if you do want to do some registry lookups on a new axis.

Utiilities, eh. In a lot of cases, it would be better to use a dictionary. Hey, what if the registry *was* a dictionary?! ... oh wait. Nevermind.

As far as doing the ZCA "full justice", I wouldn't bother writing this if I didn't believe the core of it had value.

My experience of ZCA

I'd comment here about my user experience of ZCA (I name me user as I do not develop zope packages but plone applications).

On interfaces
=============

- For me interfaces are mostly labels for inversion of control with the power of

- inheritance (so I can specialise an interface, tell which is more specific)

- namespace usage (so I can have shorter names)

- defining methods (used to define a perimeter for security)


I really feel like chris that the problem is that it may oblige you to do top-down design which is not easy.

Also at implementation level I had hard time with interfaces not being true python class (but using a prototype based design) which hurts me several time with unecessary complexity.

My whish list would be :

- defining a class implements an interface at site manager level (yes I know adapters are defined there to provide me with this but there are usecase where it xould be far more easy to define implements there)

- defining an interface from a class I want to be able to say soemthing like IMyClass = make_interface(MyClass). This would take back the docstring etc.. to implementation. ABC (Abstract Base Class) or mixin classes would fit in.


On adapters
===========

I disagree with Chris, adapters is a really good mechanism to make generic products.

I agree that your product have to define it's own API and mask ZCA mechanics when they are non trivial. I think zope.event is a good example of that as it use adaptation but hide it.

From the moment you had ZCA + viewlets in Plone you where able to add really generic, non crossing functionality : adding a rating system, tagging or such things. More over you can easily switch strategie with new adapters.

Of course pluggability have to be really well defined, at the good points. But too much pluggability often means over-engineering (an illness zope community seems to suffer ;-) ). It's not easy to find the right balance.

One problem of ZCA brings is that it squatters the code all around. To understand a simple functionnality you have to look at 3 or more files. I don't know what to do about it, but use grok which tends to make things more readable. I tend to think that generating interfaces from classes (the base class for 80 of use cases) may help.

Two problems that ZCA wont solve completely :

- defining a good API is alas specific to each situation, maybe the harder work, which comes from experience and good ideas. However to be fair, ZCA helps a bit there giving you the habits to thinks in term of separated functionnality, but does not prevent you from making a poor API.

- reusability is hard. I think reusability in Zope community is very good but this has a cost in term of complexity. It's even more complexe when you're products tend to satisfy multiple use case as it is often the case with products comming from integrators. I agree with optilude that common design patterns can help there, but a product that is flexible and configurable will allways be harder to understand than one which is not.


My whish list for ZCA :

- more diagnosis on error messages (maybe only in debug mode). Eg. if I seek for a multiadapter but the name is bad, it shall tell : "fail to get adapter from <A instance at xxx> to IMine with name 'foo'. However I got an adapter to IMine with name='spam', I got an adapter to IOther with name 'foo'". Also Failing in an adatper __init__ shall give a more explicit message.

- more introspection mechanism to registry, a user interface to fix bad registrations eg. My idea here is that you can handle complexity when you have tools for it.

- caching of adapter registry to speed up zope startup : if none of my zcml changes, take back last computed registry. Differential uupdate when some zcml changes or is added would be good but maybe far more complex to implement

- find mechanism to handle easily registration in local site manager registry. I think lot of products tends to declare eg. adapters that belongs to local site manager in global registry because it's far easier to declare. CMF generic setup maybe a good source of inspiration here.


This is a long post, I shall have made on mailing list, but as I'am not a regular follower I prefer to post here in this more intimate discussion :-)

On overuse of plug points

You make a good point that everything shouldn't have an interface. It makes it look as if each one is a public hook point. That's the sort of thing that keeps Plone from having a well-defined API and thus keeps us from changing anything to move forward. Fewer interfaces, each actually intended as a plug point, would be better for documentation purposes.

And I'd like to add something to your point about adaptation not aiding comprehesibility: adaptation works great as a plugin registry, and it's nice to have a common syntax for it, but it makes static analysis really hard. It's much harder to determine, by reading, what's going on, as code jumps off into contractual la-la land, perhaps affected by ZCML files not even belonging to the package in question. Of course, this is the purpose of plug points, but using adaptation when just a function call would do makes code needlessly hard to follow.

Third, your suggestion of not binding adaptation to the machinery of interfaces reminds me a lot of the old PyProtocols. :-)