Efficient Handling of Large EM Maps
Tomograms are large files that require efficient handling for interactive
visualization.
Typical dimensions 2048 by 2048 by 100 (400 Mbytes).
The latest detectors produce 4096 by 4096 micrographs (~2 Gbyte tomograms).
Computing limitations
- hard disk bandwidth (~1 minute to read a gigabyte).
- graphics rendering speed (256³ for interactive movement).
- physical memory (typical desktop computers, ~2 Gbytes)
- computation speed (contour calculation, denoising data, ...).
Chimera reads only needed subregions and subsamples into memory.
Requires high performance file access.
New map file format
Developed prototype new map file format based on HDF that
enables high performance with large tomograms.
Collaboration with NCMI (Wah Chiu), EM Databank (Kim Henrick, EBI) and
PDB (Helen Berman, John Westbrook).
New format capabilities:
- Subsampled maps on disk for fast reading at lower resolution.
- Chunked volume array layout on disk for fast submap reading.
- Copies with alternate axis ordering for fast axis to yz and xz planes.
- Compression of data chunks for masks (0/1 values).
- Multiple optimized data copies in single file for ease of use.
- Remember orientation for subregions extracted from maps.
- Record rotated orientation (also symmetry matrices for single particle EM).
- Unsigned 8-bit data values (not supported in MRC/CCP4 formats).
Example speed increases with new format
|
|
Test data:
Rat cortex. 2048³ map, 8 Gbytes.
Physical size 50 x 50 x 100 um.
Voxel size 25 x 25 x 50 nm (~2 ribosomes).
Serial block face scanning EM.
|
- Flip through 1K by 1K subsampled planes ~10 frames per second instead
of full resolution 4K by 4K planes (higher than screen resolution) at ~1
frame per second.
- Show low resolution view of large 1 Gbyte map in 1 second instead of
1 minute.
- Show yz slice planes in 3 seconds instead of ~3 minutes per plane for
rat cortex 2048³ map.
- Compress 5 masks of cyto-toxic T-cell map to ~5 Mbytes from uncompressed
1.5 Gbytes. Allows 300x faster read from disk.
Summary of New Chimera Tools for EM Tomography
- Plane display (gray scale).
- Planes at any orientation, TomoPlane extension (Baumeister lab).
- Resampling: extract subregions not aligned with data axes.
- IMOD segmentation display.
- Masking using segmentation surfaces.
- Measuring volumes, splitting segmentation objects.
- Large file handling: progress messages and cancel loading.
