25. Upgrade-steps – Mastering Plone 6 development

Upgrade-steps

25. Upgrade-steps#

Backend chapter

Get the code: collective/ploneconf.site

git checkout dexterity_upgrade_steps

More info in The code for the training

In this part we will:

  • Write code to update, create or move content

  • Create custom catalog indexes

  • Query the catalog for them,

  • Enable more default features for our type

Upgrade step

Upgrade step#

25.1. Upgrade steps#

You recently changed existing content, when you added the behavior ploneconf.featured or when you turned talks into events in the chapter Turning Talks into Events.

When projects evolve you sometimes want to modify various things while the site is already up and brimming with content and users. Upgrade steps are pieces of code that run when upgrading from one version of an add-on to a newer one. They can do just about anything. We will use an upgrade step to enable the new behavior instead of reinstalling the add-on.

We will create an upgrade step that:

  • runs the typeinfo step (i.e. loads the GenericSetup configuration stored in profiles/default/types.xml and profiles/default/types/... so we don't have to reinstall the add-on to have our changes from above take effect) and

  • cleans up existing talks that might be scattered around the site in the early stages of creating it. We will move all talks to a folder talks (unless they already are there).

Upgrade steps can be registered in their own ZCML file to prevent cluttering the main configure.zcml. Create the new upgrades.zcml include it in our configure.zcml:

<include file="upgrades.zcml" />

You register the first upgrade-step in upgrades.zcml:

 1<configure
 2  xmlns="http://namespaces.zope.org/zope"
 3  xmlns:i18n="http://namespaces.zope.org/i18n"
 4  xmlns:genericsetup="http://namespaces.zope.org/genericsetup"
 5  i18n_domain="ploneconf.site">
 6
 7  <genericsetup:upgradeStep
 8      title="Update and cleanup talks"
 9      description="Update typeinfo and move talks to to their folder"
10      source="1000"
11      destination="1001"
12      handler="ploneconf.site.upgrades.upgrade_site"
13      sortkey="1"
14      profile="ploneconf.site:default"
15      />
16
17</configure>

The upgrade step bumps the version number of the GenericSetup profile of ploneconf.site from 1000 to 1001. The version is stored in profiles/default/metadata.xml.

Change it to

<version>1001</version>

GenericSetup now expects the code as a method upgrade_site() in the file upgrades.py. Let's create it.

 1from plone import api
 2from plone.app.upgrade.utils import loadMigrationProfile
 3
 4import logging
 5
 6default_profile = 'profile-ploneconf.site:default'
 7logger = logging.getLogger(__name__)
 8
 9
10def reload_gs_profile(context):
11    loadMigrationProfile(
12        context,
13        'profile-ploneconf.site:default',
14    )
15
16
17def upgrade_site(context=None):
18    # reload type info
19    setup = api.portal.get_tool('portal_setup')
20    setup.runImportStepFromProfile(default_profile, 'typeinfo')
21    portal = api.portal.get()
22
23    # Create the expected folder-structure
24    if 'training' not in portal:
25        training_folder = api.content.create(
26            container=portal,
27            type='Document',
28            id='training',
29            title=u'Training')
30    else:
31        training_folder = portal['training']
32
33    if 'schedule' not in portal:
34        schedule_folder = api.content.create(
35            container=portal,
36            type='Document',
37            id='schedule',
38            title=u'Schedule')
39    else:
40        schedule_folder = portal['schedule']
41    schedule_folder_url = schedule_folder.absolute_url()
42
43    if 'location' not in portal:
44        location_folder = api.content.create(
45            container=portal,
46            type='Document',
47            id='location',
48            title=u'Location')
49    else:
50        location_folder = portal['location']
51
52    if 'sponsors' not in portal:
53        sponsors_folder = api.content.create(
54            container=portal,
55            type='Document',
56            id='sponsors',
57            title=u'Sponsors')
58    else:
59        sponsors_folder = portal['sponsors']
60
61    if 'sprint' not in portal:
62        sprint_folder = api.content.create(
63            container=portal,
64            type='Document',
65            id='sprint',
66            title=u'Sprint')
67    else:
68        sprint_folder = portal['sprint']
69
70    # Find all talks
71    brains = api.content.find(portal_type='talk')
72    for brain in brains:
73        if schedule_folder_url in brain.getURL():
74            # Skip if the talk is already somewhere inside the target folder
75            continue
76        obj = brain.getObject()
77        logger.info('Moving {} to {}'.format(
78            obj.absolute_url(), schedule_folder_url))
79        # Move talk to the folder '/schedule'
80        api.content.move(
81            source=obj,
82            target=schedule_folder,
83            safe_id=True)

Note:

  • They are simple python methods, nothing fancy about them except the registration.

  • When running a upgrade-step they get the tool portal_setup passed as a argument. To make it easier to call these steps from a pdb or from other methods it is a good idea to set it as context=None and not use the argument at all but instead use portal_setup = api.portal.get_tool('portal_setup') if you need it.

  • The portal_setup tool has a method runImportStepFromProfile(). In this example it is used to load the file profiles/default/types.xml and profiles/default/types/talk.xml to enable new behaviors, views or other settings.

  • In Python we create the required folder structure if it does not exist yet making extensive use of plone.api as discussed in the chapter Programming Plone.

After restarting the site we can run the step:

On the console you should see logging messages like:

INFO ploneconf.site.upgrades Moving http://localhost:8080/Plone/old-talk1 to http://localhost:8080/Plone/schedule

Alternatively you also select which upgrade steps to run like this:

  • In the ZMI go to portal_setup

  • Go to the tab Upgrades

  • Select ploneconf.site from the dropdown and click Choose profile

  • Run the upgrade step.

See also

https://5.docs.plone.org/develop/addons/components/genericsetup.html#upgrade-steps

Note

Upgrading from an older version of Plone to a newer one also runs upgrade steps from the package plone.app.upgrade. You should be able to upgrade a clean site from 2.5 to 5.0 with one click.

Find the upgrade steps in plone/plone.app.upgrade

25.2. Browserlayers#

Note

This section is only relevant for Plone Classic since Volto does not use Viewlets or BrowserViews.

A browserlayer is a marker on the request. Browserlayers allow us to easily enable and disable views and other site functionality based on installed add-ons and themes.

Since we want the features we write only to be available when ploneconf.site actually is installed we can bind them to a browserlayer.

Our package already has a browserlayer (added by bobtemplates.plone). See interfaces.py:

1# -*- coding: utf-8 -*-
2"""Module where all interfaces, events and exceptions live."""
3
4from zope.publisher.interfaces.browser import IDefaultBrowserLayer
5from zope.interface import Interface
6
7
8class IPloneconfSiteLayer(IDefaultBrowserLayer):
9    """Marker interface that defines a browser layer."""

It is enabled by GenericSetup when installing the package since it is registered in the profiles/default/browserlayer.xml

<?xml version="1.0" encoding="UTF-8"?>
<layers>
  <layer
      name="ploneconf.site"
      interface="ploneconf.site.interfaces.IPloneconfSiteLayer"
      />
</layers>

You should bind all your custom BrowserViews and Viewlets to it.

Here is an example using the talklistview from views_3.

<browser:page
    name="talklistview"
    for="*"
    layer="..interfaces.IPloneconfSiteLayer"
    class=".views.TalkListView"
    template="templates/talklistview.pt"
    permission="zope2.View"
    />

Note the relative Python path interfaces.IPloneconfSiteLayer. It is equivalent to the absolute path ploneconf.site.interfaces.IPloneconfSiteLayer.

See also

https://5.docs.plone.org/develop/plone/views/layers.html

25.3. Add catalog indexes#

In the talklistview we had to get the full objects to access some of their attributes. That is OK if we don't have many objects and they are light dexterity objects. If we had thousands of objects this might not be a good idea.

Note

Is is about 10 times slower to get the full objects instead of only using the resutls of a search! For 3000 objects that can make a difference of 2 seconds.

Instead of loading them all into memory we will use catalog indexes and metadata to get the data we want to display.

Add the new indexes to profiles/default/catalog.xml

 1<?xml version="1.0"?>
 2<object name="portal_catalog">
 3  <index name="featured" meta_type="BooleanIndex">
 4    <indexed_attr value="featured"/>
 5  </index>
 6  <index name="type_of_talk" meta_type="FieldIndex">
 7    <indexed_attr value="type_of_talk"/>
 8  </index>
 9  <index name="speaker" meta_type="FieldIndex">
10    <indexed_attr value="speaker"/>
11  </index>
12  <index name="audience" meta_type="KeywordIndex">
13    <indexed_attr value="audience"/>
14  </index>
15  <index name="room" meta_type="FieldIndex">
16    <indexed_attr value="room"/>
17  </index>
18
19  <column value="featured" />
20  <column value="type_of_talk" />
21  <column value="speaker" />
22  <column value="audience" />
23  <column value="room" />
24</object>

This adds new indexes for the three fields we want to show in the listing. Note that audience is a KeywordIndex because the field is multi-valued, but we want a separate index entry for every value in an object.

The column .. entries allow us to display the values of these indexes in the tableview of collections.

See also

https://5.docs.plone.org/develop/plone/searching_and_indexing/indexing.html

Note

The new indexes are still empty. We'll have to reindex them. To do so by hand go to http://localhost:8080/Plone/portal_catalog/manage_catalogIndexes, select the new indexes and click Reindex. We could also rebuild the whole catalog by going to the Advanced tab and clicking Clear and Rebuild. For large sites that can take a long time.

We could also write an upgrade step to enable the catalog indexes and reindex all talks:

def add_some_indexes(setup):
    setup.runImportStepFromProfile(default_profile, 'catalog')
    for brain in api.content.find(portal_type='talk'):
        obj = brain.getObject()
        obj.reindexObject(idxs=[
          'type_of_talk',
          'speaker',
          'audience',
          'room',
          'featured',
          ])

Todo

  1. Adapt the TalkListView in Volto to not use fullobjects. Instead either pass a list of metadata-fields or use metadata_fields=_all to get the equivalent of brains as documented in {ref}`plone6docs:retrieving-additional-metadata`.

  2. Adapt the colored audience-blocks in TalkView in Volto to use the custom index to find all talks for that audience. The Volto search needs to support all indexes dynamically for that to work!

    <Link
  className={`ui label ${color}`}
  to={`/search?portal_type=talk&audience=${audience}`}
  key={audience}
>
  {audience}
</Link>

25.4. Query for custom indexes#

The new indexes behave like the ones that Plone has already built in:

>>> (Pdb) from Products.CMFCore.utils import getToolByName
>>> (Pdb) catalog = getToolByName(self.context, 'portal_catalog')
>>> (Pdb) catalog(type_of_talk='Keynote')
[<Products.ZCatalog.Catalog.mybrains object at 0x10737b9a8>, <Products.ZCatalog.Catalog.mybrains object at 0x10737b9a8>]
>>> (Pdb) catalog(audience=('Advanced', 'Professional'))
[<Products.ZCatalog.Catalog.mybrains object at 0x10737b870>, <Products.ZCatalog.Catalog.mybrains object at 0x10737b940>, <Products.ZCatalog.Catalog.mybrains object at 0x10737b9a8>]
>>> (Pdb) brain = catalog(type_of_talk='Keynote')[0]
>>> (Pdb) brain.speaker
u'David Glick'

If you use the classic frontend with the BrowserView talklistview you can now use these new indexes to improve it so we don't have to wake up the objects anymore.

Instead you can use the brains' new attributes.

 1class TalkListView(BrowserView):
 2    """ A list of talks
 3    """
 4
 5    def talks(self):
 6        results = []
 7        brains = api.content.find(context=self.context, portal_type='talk')
 8        for brain in brains:
 9            results.append({
10                'title': brain.Title,
11                'description': brain.Description,
12                'url': brain.getURL(),
13                'audience': ', '.join(brain.audience or []),
14                'type_of_talk': brain.type_of_talk,
15                'speaker': brain.speaker,
16                'room': brain.room,
17                'uuid': brain.UID,
18                })
19        return results

The template does not need to be changed and the result in the browser did not change either. But when listing a large number of objects the site will now be faster since all the data you use comes from the catalog and the objects do not have to be loaded into memory.

Todo

Explain when having custom indexes and metadata makes sense with Volto.

25.5. Exercise 1#

Note

This exercise is only relevant for Plone Classic.

In fact we could now simplify the view even further by only returning the brains.

Modify TalkListView to return only brains and adapt the template to these changes. Remember to move ', '.join(brain.audience or []) into the template.

Solution

Here is the class:

1class TalkListView(BrowserView):
2    """ A list of talks
3    """
4
5    def talks(self):
6        return api.content.find(context=self.context, portal_type='talk')

Here is the template:

 1<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"
 2      metal:use-macro="context/main_template/macros/master"
 3      i18n:domain="ploneconf.site">
 4<body>
 5  <metal:content-core fill-slot="content-core">
 6
 7  <table class="listing"
 8         id="talks"
 9         tal:define="brains python:view.talks()">
10    <thead>
11      <tr>
12        <th>Title</th>
13        <th>Speaker</th>
14        <th>Audience</th>
15        <th>Room</th>
16      </tr>
17    </thead>
18    <tbody>
19      <tr tal:repeat="brain brains">
20        <td>
21          <a href=""
22             tal:attributes="href python:brain.getURL();
23                             title python:brain.Description"
24             tal:content="python:brain.Title">
25             The 7 sins of Plone development
26          </a>
27        </td>
28        <td tal:content="python:brain.speaker">
29            Philip Bauer
30        </td>
31        <td tal:content="python:', '.join(brain.audience or [])">
32            Advanced
33        </td>
34        <td tal:content="python:brain.room">
35            Room 101
36        </td>
37      </tr>
38      <tr tal:condition="not:brains">
39        <td colspan=4>
40            No talks so far :-(
41        </td>
42      </tr>
43    </tbody>
44  </table>
45
46  </metal:content-core>
47</body>
48</html>

25.6. Add collection criteria#

In chapter Behaviors you already added the field featured as a querystring-criterion.

To be able to search content using these new indexes in collections and listing blocks we need to register them as well.

As with all features make sure you only do this if you really need it!

Add criteria for audience, type_of_talk and speaker to the file profiles/default/registry/querystring.xml.

 1<?xml version="1.0" encoding="UTF-8"?>
 2<registry xmlns:i18n="http://xml.zope.org/namespaces/i18n"
 3          i18n:domain="plone">
 4
 5  <records interface="plone.app.querystring.interfaces.IQueryField"
 6           prefix="plone.app.querystring.field.featured">
 7    <value key="title">Featured</value>
 8    <value key="enabled">True</value>
 9    <value key="sortable">False</value>
10    <value key="operations">
11      <element>plone.app.querystring.operation.boolean.isTrue</element>
12      <element>plone.app.querystring.operation.boolean.isFalse</element>
13    </value>
14    <value key="group" i18n:translate="">Metadata</value>
15  </records>
16
17  <records interface="plone.app.querystring.interfaces.IQueryField"
18           prefix="plone.app.querystring.field.audience">
19    <value key="title">Audience</value>
20    <value key="description">A custom speaker index</value>
21    <value key="enabled">True</value>
22    <value key="sortable">False</value>
23    <value key="operations">
24      <element>plone.app.querystring.operation.string.is</element>
25      <element>plone.app.querystring.operation.string.contains</element>
26    </value>
27    <value key="group" i18n:translate="">Metadata</value>
28  </records>
29
30  <records interface="plone.app.querystring.interfaces.IQueryField"
31           prefix="plone.app.querystring.field.type_of_talk">
32    <value key="title">Type of Talk</value>
33    <value key="description">A custom index</value>
34    <value key="enabled">True</value>
35    <value key="sortable">False</value>
36    <value key="operations">
37      <element>plone.app.querystring.operation.string.is</element>
38      <element>plone.app.querystring.operation.string.contains</element>
39    </value>
40    <value key="group" i18n:translate="">Metadata</value>
41  </records>
42
43  <records interface="plone.app.querystring.interfaces.IQueryField"
44           prefix="plone.app.querystring.field.speaker">
45    <value key="title">Speaker</value>
46    <value key="description">A custom index</value>
47    <value key="enabled">True</value>
48    <value key="sortable">False</value>
49    <value key="operations">
50      <element>plone.app.querystring.operation.string.is</element>
51      <element>plone.app.querystring.operation.string.contains</element>
52    </value>
53    <value key="group" i18n:translate="">Metadata</value>
54  </records>
55
56</registry>

See also

https://5.docs.plone.org/develop/plone/functionality/collections.html#add-new-collection-criteria-new-style-plone-app-collection-installed

25.7. Add versioning through GenericSetup#

You already enabled the versioning behavior on the content type. It allows you to specify if versioning should be enabled for each individual object instead of using a default-setting per content type. See profiles/default/types/talk.xml:

1<property name="behaviors">
2 <element value="plone.dublincore"/>
3 <element value="plone.namefromtitle"/>
4 <element value="plone.versioning" />
5 <element value="ploneconf.featuered"/>
6</property>

You still need to configure the versioning policy and a diff view for talks.

Add new file profiles/default/repositorytool.xml

1<?xml version="1.0"?>
2<repositorytool>
3  <policymap>
4    <type name="talk">
5      <policy name="at_edit_autoversion"/>
6      <policy name="version_on_revert"/>
7    </type>
8  </policymap>
9</repositorytool>

Add new file profiles/default/diff_tool.xml

1<?xml version="1.0"?>
2<object>
3  <difftypes>
4    <type portal_type="talk">
5      <field name="any" difftype="Compound Diff for Dexterity types"/>
6    </type>
7  </difftypes>
8</object>

25.8. Summary#

  • You wrote your first upgrade step to move the talks around: yipee!

  • Some fields are indexed in the catalog allowing to search for these and making listings much faster

  • Versioning for Talks is now properly configured