Opened 2 hours ago

Last modified 2 hours ago

#20558 assigned enhancement

Avoid copying volumes to bytes when saving sessions

Reported by: Tom Goddard Owned by: Tom Goddard
Priority: moderate Milestone:
Component: Volume Data Version:
Keywords: Cc:
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description

When ChimeraX includes volume data in a session file it first converts the 3D array into bytes. This makes a full copy of the data in memory which doubles the memory use and sometimes results in out of memory errors, especially on Windows, such as ticket #20552.

In ChimeraX 1.12 (possibly older versions) our session saving msgpack code can handle numpy arrays so we could avoid converting to a bytes object and avoid doubling the memory.

Change History (1)

comment:1 by Tom Goddard, 2 hours ago

I tried saving the flattened numpy array instead of bytes in the ChimeraX 1.13 daily build and it saved and restored correctly. The flattened version is just a view on the original data so it should not increase memory use. The session restored in ChimeraX 1.12. It failed to restore in ChimeraX 1.11.1 with an error about missing SceneAnimation class. There was no use of scenes or animations in the test session but I suspect if the new ChimeraX did not put in a dependency on SceneAnimation it would still fail because I think our 1.11 release msgpack did not handle numpy arrays. We don't guarantee forward compatibility so that may be ok.

I would be worried that we are about to switch from numpy 1 to numpy 2 and possibly msgpack will have trouble restoring an numpy 1 array with numpy 2. Need to test that so that backward compatibility of sessions is not broken.

Note: See TracTickets for help on using tickets.