readcif – a C++11 CIF and mmCIF parser

Author:Greg Couch Organization:RBVI, University of California at San Francisco Contact:gregc@cgl.ucsf.edu Copyright:© Copyright 2014 by the Regents of the University of California. All Rights reserved. Last modified:2014-6-17

readcif is a C++11 library for quickly extracting data from mmCIF and CIF files. It fully conforms to the CIF 1.1 standard for data files, and can be easily extended to handle CIF dictionaries. In addition, it supports stylized PDBx/mmCIF files for even quicker parsing.

License

The readcif library is available with an open source license:

Copyright © 2014 The Regents of the University of California. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions, and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Redistributions must acknowledge that this software was originally developed by the UCSF Resource for Biocomputing, Visualization, and Informatics with support from the National Institute of General Medical Sciences, grant P41-GM103311.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OF THE UNIVERSITY OF CALIFORNIA BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Usage

CIF files are essentially a text version of a database. Each table in the database corresponds to a category, and the columns of the table are labelled with tags, and the rows contain the values.

readcif provides a base class, CIFFile, that should be subclassed to implement an application’s specific needs. Virtual functions are used to support CIF keywords, that way the application can choose what to do if there is more than one data block or handle dictionaries. And callback functions are used to extract the data the application wants from a category. Finally, the category callback functions need to provide a set of callback functions to parse the value of interesting tags.

So, in pseudo-code, an application’s parser would look like:

class ExtractCIF: public readcif::CIFFile {
public:
    ExtractCIF() {
        initialize category callbacks
    }
    in each category callback:
        create parse value callback vector for interesting tags
        while (parse_row(parse_value_callbacks))
            continue;
};

and be used by:

ExtractCIF extract;

const char* whole_file = ....
extract.parse_file(filename)

See the associated readcif_example.cpp file for a working example that reads a subset of the atom_site data from a PDB mmCIF file.

Stylized PDBx/mmCIF files

If a CIF file is known to use PDBx/mmCIF stylized formatting, then parsing can be up to 4 times faster. set_PDB_style() turns on and off stylized parsing, and is reset every time CIFFile::parse() is called. Stylized PDBx/mmCIF file parsing may be switched on and off at any time, e.g., within a particular category callback function.

The example code turns on stylized parsing if the audit_conform.dict_name is mmcif_pdbx.dic and audit_conform.dict_version is greater than 4. This is hack that is needed until the PDB adds an explicit tag and value near the beginning of a CIF file that indicates that its tables are stylized.

PDBx/mmCIF Styling

PDBx/mmCIF files are formatted for fast parsing. readcif expects the following syntax for stylized files:

Outside of a data table:

1. CIF keywords and data tags only appear immediately after an ASCII newline.
2. CIF keywords are in lowercase.
3. Data tags are case sensitive (category names and item names are mixed-case as specified in mmcif_pdbx.dic).

Inside a data table:

1. If the data values for each row can’t fit on one line (due to a multiline string), then the first row is split into multiple lines.
2. All columns are left-aligned.
3. All rows have trailing spaces so they are the same length.
4. Rows are terminated by a comment line.

C++ API

All of the public symbols are in the readcif namespace.

type StringVector

A std::vector of std::string’s.

int is_whitespace(char c)

is_whitespace and is_not_whitespace are inline functions to determine if a character is CIF whitespace or not. They are similar to the C/C++ standard library’s isspace function, but only recognize ASCII HT (9), LF (10), CR (13), and SPACE (32) as whitespace characters. They are not inverses because ASCII NUL (0) is both not is_whitespace and not is_not_whitespace.

int is_not_whitespace(char c)

See is_whitespace().

float str_to_float(const char* s)

Non-error checking inline function to convert a string to a floating point number. It is similar to the C/C++ standard library’s strtof function, but does not support scientific notation, and is about twice as fast.

int str_to_int(const char* s)

Non-error inline function to convert a string to an integer. It is similar to the C/C++ standard library’s atof function.

class CIFFile

The CIFFile is designed to be subclassed by an application to extract the data the application is interested in.

Public section:

type ParseCategory

A typedef for std::function<void (bool in_loop)>.

void register_category(const std::string& category, ParseCategory callback, const StringVector& dependencies=StringVector())

Register a callback function for a particular category.

Parameters:

• category – name of category
• callback – function to retrieve data from category
• dependencies – a list of categories that must be parsed before this category.

A null callback function, removes the category. Dependencies must be registered first. A category callback function can find out which category it is processing with category().

void parse_file(const char* filename)

Parameters filename:  Name of file to be parsed

If possible, memory-map the given file to get the buffer to hand off to parse(). On POSIX systems, files whose size is a multiple of the system page size, have to be read into an allocated buffer instead.

void parse(const char* buffer)

Parse the input and invoke registered callback functions

Parameters buffer:  Null-terminated text of the CIF file

The text must be terminated with a null character. A common technique is to memory map a file and pass in the address of the first character. The whole file is required to simplify backtracking since data tables may appear in any order in a file. Stylized parsing is reset each time parse() is called.

void set_PDB_style(bool stylized)

Turn on and off PDBx/mmCIF stylized parsing

Parameters stylized:  true to use PDBx/mmCIF stylized parsing

Indicate that CIF file follows the PDBx/mmCIF style guide and that the style can be followed to speed up lexical analysis of the CIF file. Specifically, that keywords are in lowercase, and that all keywords and tags are at the beginning of a line.

bool PDB_style() const

Return if the PDB_style flag is set. See set_PDB_style().

void set_PDB_fixed_columns(bool fc)

Turn on and off PDBx/mmCIF fixed column width parsing

Parameters stylized:  true to use PDBx/mmCIF stylized parsing

Indicate that CIF file follows the PDBx/mmCIF style guide and that the style can be followed to speed up lexical analysis of the CIF file. Specifically, the category’s data layout has each record of data on a single line and the columns of data are left-justified, and are of are fixed width. This option must be set in each category callback that is needed. This option is ignored if :cpp:`PDB_style` is false. This is not a global option because there is no reliable way to detect if the preconditions are met for each record without losing all of the speed advantages.

bool PDB_fixed_columns() const

Return if the PDB_fixed_columns flag is set. See set_PDB_fixed_columns().

int get_column(const char* tag, bool required=false)

Parameters:

• tag – column tag to search for
• required – true if tag is required

Search the current categories tags to figure out which column the tag corresponds to. If the tag is not present, then -1 is returned unless it is required, then an error is thrown.

type ParseValue

typedef std::function<void (const char* start, const char* end)> ParseValue;

type ParseValues

typedef std::vector<ParseColumn> ParseValues;

bool parse_row(ParseValues& pv)

Parse a single row of a table

Parameters pv:The per-column callback functions Returns:if a row was parsed

The category callback functions should call parse_row(): to parse the values for columns it is interested in. If in a loop, parse_row(): should be called until it returns false, or to skip the rest of the values, just return from the category callback. The first time parse_row() is called for a category, pv will be sorted in ascending order. Columns with negative offsets are skipped.

const std::string& version()

Returns:the version of the CIF file if it is given

For mmCIF files it is typically empty.

const std::string& category()

Returns:the category that is currently being parsed

Only valid within a ParseCategory callback.

const std::string& block_code()

Returns:the data block code that is currently being parsed

Only valid within a ParseCategory callback and finished_parse().

const StringVector& tags()

Returns:the set of column tags for the current category

Only valid within a ParseCategory callback.

std::runtime_error error(const std::string& text)

Parameters text:  the error message Returns:a exception with ” on line #” appended Rtype:std::runtime_error

Localize error message with the current line number within the input. # is the current line number.

Protected section:

void data_block(const std::string& name)

Parameters name:  name of data block

data_block is a virtual function that is called whenever a new data block is found. Defaults to being ignored. Replace in subclass if needed.

void save_frame(const std::string& code)

Parameters code:  the same frame code

save_fame is a virtual function that is called when a save frame header or terminator is found. It defaults to throwing an exception. It should be replaced if the application were to try to parse a dictionary.

void global_block()

global_block is a virtual function that is called whenever the global_ keyword is found. It defaults to throwing an exception. In CIF files, the global_ keyword is reserved, but unused. However, some CIF-like files, e.g., the CCP4 monomer library, use the global_ keyword.

void reset_parse()

reset_parse is a virtual function that is called whenever the parse function is called. For example, PDB stylized parsing can be turned on here.

void finished_parse()

finished_parse is a virtual function that is called whenever the parse function has successfully finished parsing.