Bundle Example: Read a New File Format

This example describes how to create a ChimeraX bundle that allows ChimeraX to open data files in XYZ format, which is a simple format containing only information about atomic types and coordinates.

The steps in implementing the bundle are:

1. Create a bundle_info.xml containing information about the bundle,

2. Create a Python package that interfaces with ChimeraX and implements the file-reading functionality, and

3. Install and test the bundle in ChimeraX.

The final step builds a Python wheel that ChimeraX uses to install the bundle. So if the bundle passes testing, it is immediately available for sharing with other users.

Source Code Organization

The source code for this example may be downloaded as a zip-format file containing a folder named tut_read. Alternatively, one can start with an empty folder and create source files based on the samples below. The source folder may be arbitrarily named, as it is only used during installation; however, avoiding whitespace characters in the folder name bypasses the need to type quote characters in some steps.

Sample Files

The files in the tut_read folder are:

• tut_read - bundle folder

  • bundle_info.xml - bundle information read by ChimeraX

  • src - source code to Python package for bundle

    • __init__.py - package initializer and interface to ChimeraX

    • io.py - source code to read XYZ format files

The file contents are shown below.

bundle_info.xml

bundle_info.xml is an eXtensible Markup Language format file whose tags are listed in Bundle Information XML Tags. While there are many tags defined, only a few are needed for bundles written completely in Python. The bundle_info.xml in this example is similar to the one from the Bundle Example: Add a Tool example with changes highlighted. For explanations of the unhighlighted sections, please see Bundle Example: Hello World, Bundle Example: Add a Command and Bundle Example: Add a Tool.

<!--
ChimeraX bundle names must start with "ChimeraX-"
to avoid clashes with package names in pypi.python.org.
When uploaded to the ChimeraX toolshed, the bundle
will be displayed without the ChimeraX- prefix.
-->

<BundleInfo name="ChimeraX-TutorialReadFormat"
	    version="0.1" package="chimerax.tut_read"
  	    minSessionVersion="1" maxSessionVersion="1">

  <!-- Additional information about bundle source -->
  <Author>UCSF RBVI</Author>
  <Email>chimerax@cgl.ucsf.edu</Email>
  <URL>https://www.rbvi.ucsf.edu/chimerax/</URL>

  <!-- Synopsis is a one-line description
       Description is a full multi-line description -->
  <Synopsis>Example for reading XYZ format files</Synopsis>
  <Description>Example code for implementing ChimeraX bundle.

Implements capability for reading XYZ format files and creating
ChimeraX atomic structures.
  </Description>

  <!-- Categories is a list where this bundle should appear -->
  <Categories>
    <Category name="General"/>
  </Categories>

  <!-- Dependencies on other ChimeraX/Python packages -->
  <Dependencies>
    <Dependency name="ChimeraX-Core" version="~=1.1"/>
  </Dependencies>

    <!-- Register XYZ format as one of the supported input file formats -->
  <Providers manager="data formats">
    <Provider name="XYZ" suffixes=".xyz" category="Molecular structure"
		reference_url="https://en.wikipedia.org/wiki/XYZ_file_format"
		encoding="utf-8" mime_types="chemical/x-xyz" />
  </Providers>

  <Providers manager="open command">
    <Provider name="XYZ" />
  </Providers>

  <Classifiers>
    <!-- Development Status should be compatible with bundle version number -->
    <PythonClassifier>Development Status :: 3 - Alpha</PythonClassifier>
    <PythonClassifier>License :: Freeware</PythonClassifier>
  </Classifiers>

</BundleInfo>

The BundleInfo, Synopsis and Description tags are changed to reflect the new bundle name and documentation (lines 8-10 and 17-24).

The Providers sections on lines 36 through 45 use the Manager/Provider protocol to inform the “data formats” manager about the XYZ format, and the “open command” manager that this bundle can open XYZ files.

The attributes usable with the “data formats” manager are described in detail in Defining a File/Data Format. Note that most formats have a longer official name than “XYZ” and therefore most formats will also specify nicknames and synopsis attributes, whereas they are unneeded in this example.

Similarly, the “open command” attributes are described in detail in Opening Files. It is typical that the only attribute specified is name.

src

src is the folder containing the source code for the Python package that implements the bundle functionality. The ChimeraX devel command, used for building and installing bundles, automatically includes all .py files in src as part of the bundle. (Additional files may also be included using bundle information tags such as DataFiles as shown in Bundle Example: Add a Tool.) The only required file in src is __init__.py. Other .py files are typically arranged to implement different types of functionality. For example, cmd.py is used for command-line commands; tool.py or gui.py for graphical interfaces; io.py for reading and saving files, etc.

src/__init__.py

As described in Bundle Example: Hello World, __init__.py contains the initialization code that defines the bundle_api object that ChimeraX needs in order to invoke bundle functionality. ChimeraX expects bundle_api class to be derived from chimerax.core.toolshed.BundleAPI with methods overridden for registering commands, tools, etc.

# vim: set expandtab shiftwidth=4 softtabstop=4:

from chimerax.core.toolshed import BundleAPI


# Subclass from chimerax.core.toolshed.BundleAPI and
# override the method for opening files,
# inheriting all other methods from the base class.
class _MyAPI(BundleAPI):

    api_version = 1

    # Implement provider method for opening file
    @staticmethod
    def run_provider(session, name, mgr):
        # 'run_provider' is called by a manager to invoke the 
        # functionality of the provider.  Since the "data formats"
        # manager never calls run_provider (all the info it needs
        # is in the Provider tag), we know that only the "open
        # command" manager will call this function, and customize
        # it accordingly.
        #
        # The 'name' arg will be the same as the 'name' attribute
        # of your Provider tag, and mgr will be the corresponding
        # Manager instance
        #
        # For the "open command" manager, this method must return
        # a chimerax.open_command.OpenerInfo subclass instance.
        from chimerax.open_command import OpenerInfo
        class XyzOpenerInfo(OpenerInfo):
            def open(self, session, data, file_name, **kw):
                # The 'open' method is called to open a file,
                # and must return a (list of models created,
                # status message) tuple.
                from .io import open_xyz
                return open_xyz(session, data)
        return XyzOpenerInfo()


# Create the ``bundle_api`` object that ChimeraX expects.
bundle_api = _MyAPI()

The run_provider() method is called by a ChimeraX manager when it needs additional information from a provider or it needs a provider to execute a task. The session argument is a Session instance, the name argument is the same as the name attribute in your Provider tag, and the mgr argument is the manager instance. These arguments can be used to decide what to do when your bundle offers several Provider tags (to possibly several managers), but since the “data formats” manager never calls run_provider(), we can customize the routine specifically for the “open command” manager and don’t need to check the run_provider() arguments.

When called by the “open command” manager, run_provider() must return an instance of a subclass of chimerax.open_command.OpenerInfo. The methods of the class are thoroughly documented if you click the preceding link, but briefly:

1. The open() method is called to actually open/read the file and should return a (models, status message) tuple. The method’s data argument is normally an opened stream encoded as per the format’s encoding attribute (binary if omitted), but it can be a file path if certain Provider attributes were specified (most often, want_path="true").

2. If there are format-specific keyword arguments that the open command should handle, then an open_args() property should be implemented, which returns a dictionary mapping Python keyword names to Annotation subclasses. Such keywords will be passed to your open() method.

3. So long as your open() method accepts a stream, opening compressed files of your format (e.g. with additional suffixes such as .gz, .bz2) will be handled automatically. For path-based openers, such files will result in an error before your opener is called.

4. If for some reason the opened file should not appear in the file history, set in_file_history to False.

src/io.py

# vim: set expandtab shiftwidth=4 softtabstop=4:


def open_xyz(session, stream):
    """Read an XYZ file from a file-like object.

    Returns the 2-tuple return value expected by the
    "open command" manager's :py:meth:`run_provider` method.
    """
    structures = []
    line_number = 0
    atoms = 0
    bonds = 0
    while True:
        s, line_number = _read_block(session, stream, line_number)
        if not s:
            break
        structures.append(s)
        atoms += s.num_atoms
        bonds += s.num_bonds
    status = ("Opened XYZ file containing %d structures (%d atoms, %d bonds)" %
              (len(structures), atoms, bonds))
    return structures, status


def _read_block(session, stream, line_number):
    # XYZ files are stored in blocks, with each block representing
    # a set of atoms.  This function reads a single block
    # and builds a ChimeraX AtomStructure instance containing
    # the atoms listed in the block.

    # First line should be an integer count of the number of
    # atoms in the block.
    count_line = stream.readline()
    if not count_line:
        # Reached EOF, normal termination condition
        return None, line_number
    line_number += 1
    try:
        count = int(count_line)
    except ValueError:
        session.logger.error("line %d: atom count missing" % line_number)
        return None, line_number

    # Create the AtomicStructure instance for atoms in this block.
    # All atoms in the structure are placed in one residue
    # since XYZ format does not partition atoms into groups.
    from chimerax.atomic import AtomicStructure
    from numpy import array, float64
    s = AtomicStructure(session)
    residue = s.new_residue("UNK", 'A', 1)

    # XYZ format supplies the atom element type only, but
    # ChimeraX keeps track of both the element type and
    # a unique name for each atom.  To construct the unique
    # atom name, the # 'element_count' dictionary is used
    # to track the number of atoms of each element type so far,
    # and the current count is used to build unique atom names.
    element_count = {}

    # Next line is a comment line
    s.comment = stream.readline().strip()
    line_number += 1

    # import convenience function for adding atoms
    from chimerax.atomic.struct_edit import add_atom

    # There should be "count" lines of atoms.
    for n in range(count):
        atom_line = stream.readline()
        if not atom_line:
            session.logger.error("line %d: atom data missing" % line_number)
            return None, line_number
        line_number += 1

        # Extract available data
        parts = atom_line.split()
        if len(parts) != 4:
            session.logger.error("line %d: atom data malformatted"
                                 % line_number)
            return None, line_number

        # Convert to required parameters for creating atom.
        # Since XYZ format only required atom element, we
        # create a unique atom name by putting a number after
        # the element name.
        xyz = [float(v) for v in parts[1:]]
        element = parts[0]
        n = element_count.get(element, 0) + 1
        name = element + str(n)
        element_count[element] = n

        # Create atom in AtomicStructure instance 's',
        # set its coordinates, and add to residue
        atom = add_atom(name, element, residue, array(xyz, dtype=float64))

    # Use AtomicStructure method to add bonds based on interatomic distances
    s.connect_structure()

    # Return AtomicStructure instance and current line number
    return s, line_number

The open_xyz() function is called from the __init__.bundle_api.open_file() method to open an input file in XYZ format. The contents of such a file is a series of blocks, each representing a single molecular structure. Each block in an XYZ format file consists of

• a line with the number atoms in the structure,

• a comment line, and

• one line per atom, containing four space-separated fields: element type and x, y, and z coordinates.

The return value that ChimeraX expects from open_xyz() is a 2-tuple of a list of structures and a status message. The open_xyz() code simply initializes an empty list of structures (line 10) and repeatedly calls _read_block() until the entire file is read (lines 14-20). When read_block() successfully reads a block, it returns an instance of chimerax.atomic.AtomicStructure, which is added to the structure list (line 18); otherwise, it returns None which terminates the block-reading loop (lines 16-17). A status message is constructed from the total number of structures, atoms, and bonds (lines 21-22). The structure list and the status message are then returned to ChimeraX for display (line 23).

_read_block() reads and constructs an atomic structure in several steps:

1. read the number of atoms in the block (lines 32-43).

2. build an empy atomic structure to which atoms will be added (lines 45-51). The chimerax.atomic.AtomicStructure instance is created on line 50, and a chimerax.atomic.Residue instance is created on line 51. The latter is required because ChimeraX expects every atom in a structure to be part of exactly one residue in the same structure. Even though XYZ format does not support the concept of residues, a dummy one is created anyway.

3. skip the comment line (lines 61-63).

4. loop over the expected number of atoms and add them to the structure (lines 68-95). The construction of a chimerax.atomic.Atom instance is somewhat elaborate (lines 83-95). First, the atom parameters are prepared: the atomic coordinates are extracted from the input (line 87), and the atom name is constructed from the element type and an element-specific running index (lines 88-91). The Atom instance is created on line 95, using the convenience function chimerax.atomic.struct_edit.add_atom() which also adds it to the residue, and sets its coordinates.

5. XYZ format files do not have connectivity information, so no bonds are created while processing input lines. Instead, the connect_structure() method of the structure is called to deduce connectivity from interatomic distances (line 98).

6. Return success or failure to read a structure to open_xyz (line 101).

Building and Testing Bundles

To build a bundle, start ChimeraX and execute the command:

devel build PATH_TO_SOURCE_CODE_FOLDER

Python source code and other resource files are copied into a build sub-folder below the source code folder. C/C++ source files, if any, are compiled and also copied into the build folder. The files in build are then assembled into a Python wheel in the dist sub-folder. The file with the .whl extension in the dist folder is the ChimeraX bundle.

To test the bundle, execute the ChimeraX command:

devel install PATH_TO_SOURCE_CODE_FOLDER

This will build the bundle, if necessary, and install the bundle in ChimeraX. Bundle functionality should be available immediately.

To remove temporary files created while building the bundle, execute the ChimeraX command:

devel clean PATH_TO_SOURCE_CODE_FOLDER

Some files, such as the bundle itself, may still remain and need to be removed manually.

Building bundles as part of a batch process is straightforward, as these ChimeraX commands may be invoked directly by using commands such as:

ChimeraX --nogui --exit --cmd 'devel install PATH_TO_SOURCE_CODE_FOLDER exit true'

This example executes the devel install command without displaying a graphics window (--nogui) and exits immediately after installation (exit true). The initial --exit flag guarantees that ChimeraX will exit even if installation fails for some reason.

Distributing Bundles

With ChimeraX bundles being packaged as standard Python wheel-format files, they can be distributed as plain files and installed using the ChimeraX toolshed install command. Thus, electronic mail, web sites and file sharing services can all be used to distribute ChimeraX bundles.

Private distributions are most useful during bundle development, when circulation may be limited to testers. When bundles are ready for public release, they can be published on the ChimeraX Toolshed, which is designed to help developers by eliminating the need for custom distribution channels, and to aid users by providing a central repository where bundles with a variety of different functionality may be found.

Customizable information for each bundle on the toolshed includes its description, screen captures, authors, citation instructions and license terms. Automatically maintained information includes release history and download statistics.

To submit a bundle for publication on the toolshed, you must first sign in. Currently, only Google sign in is supported. Once signed in, use the Submit a Bundle link at the top of the page to initiate submission, and follow the instructions. The first time a bundle is submitted to the toolshed, it is held for inspection by the ChimeraX team, which may contact the authors for more information. Once approved, all subsequent submissions of new versions of the bundle are posted immediately on the site.

What’s Next

• Bundle Example: Hello World

• Bundle Example: Add a Command

• Bundle Example: Add a Tool (previous topic)

• Bundle Example: Read a New File Format (current topic)

• Bundle Example: Save a New File Format (next topic)

• Bundle Example: Fetch from Network Database

• Bundle Example: Define a Chemical Subgroup Selector

• Bundle Example: Define Presets