id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	blockedby	blocking	notify_on_close	platform	project
4073	Need reliable linking for python C++ modules using libarrays	Tom Goddard	Tom Goddard	"About 35 modules of ChimeraX have C++ code that uses the arrays module for parsing numpy array Python arguments.  Loading those C++ modules will fail if ""import arrays; arrays.load_libarrays()"" is not done first to dlopen libarrays.  Currently we just call load_libarrays() in two modules (geometry and atomic) and pray that no other module is imported earlier.  This burned us in bug #4068 where the Sphinx documentation imports each module with no specified order.

In the future it would be interesting to allow Python developers to simply import a chimerax module without running the ChimeraX app and use whatever functionality they need.  For that to work we can't rely on load_libarrays() being called in one specific place in the application.  Every place a C++ module using libarrays is used must assure load_libarrays() is called.

The normal solution for runtime linking is that _map.dylib that needs libarrays.dylib would just reference it with either @rpath/libarrays.dylib or a relative path ../arrays/libarrays.dylib.  If using @rpath then the executable (python) needs to define rpath or an environment variable LD_LIBRARY_PATH defines it.  Neither of those rpath definitions are convenient for using a Python module.  The relative path could work as long as all Python bundles are installed in the same place.  With Toolshed installing updates in the users home directory instead of in the app, that would break this assumption.  I don't think Toolshed updates of individual ChimeraX bundles is going to be useful.  But relying on the relative paths built into the C++ libraries seems fragile.

A more robust solution would be to have a wrapper Python module that calls load_libarray() before importing the actual C++ module.  For instance, for the _map C++ libary have also a map_cpp.py file that calls load_libarray() then imports all of _map.  All code would use map_cpp.  Only map_cpp would directly import _map.  This is a bit painful to have an ugly shim library just to assure runtime linking works but seems the most robust.  The arrays C++ library being used by many Python modules is an unusual organization for python and runtime linking is always somewhat nightmarish for large software since any changes in directory structure of the distributed code are likely to break it.
"	enhancement	closed	moderate		Infrastructure		fixed		Eric Pettersen Greg Couch				all	ChimeraX