id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	blockedby	blocking	notify_on_close	platform	project
3845	Handle low bandwidth in VR meetings	Tom Goddard	Tom Goddard	"Want VR meetings to work for participants with lower bandwidth network connections and also handle changes of bandwidth in the middle of a meeting.

Today Phil Cruz, Victor Kramer and I had a VR meeting looking at sars-cov-2 spike and binding proteins.  It worked smoothly for about 20 minutes and then Phil's bandwidth seemed to drop, his hand and head positions seen by me and Victor updated slowly and he did not see us moving.  We were talking at the same time by Zoom audio an his audio was often garbled.  Phil quit the meeting with meeting close then tried to rejoin.  It took about 2 minutes to receivec the 25 Mbyte session while the first time he joined it took about 10 seconds.  After he saw the molecules it appears he saw the changes Victor and I made as we looked at different binding proteins a few minutes after we made those changes.  During this time I did not see Phil's head or hands but Victor did see them.  I had error messages due to code that was checking for null bytes in messages, the debug code itself had the error, no null bytes, which may have caused me to  not see his head/hands.

It appears what happened is that messages were being sent to Phil faster than the network connection could provide.  Since there is unlimited buffering the backlog of commands sent to Phil just grew and grew.  Likewise his upload speed was probably even slower than download since we had trouble understanding his Zoom audio but he heard us fine.  So his meeting messages updating head and hand positions 80-90 times per second were probably also even more backlogged, so Victor and I saw very little motion.  (I could sometimes see model moves Phil had done even though I could not see his head and hands.)

After Phil rejoined the 2 minutes while he was receiving the session probably buffered up 2 minutes worth of me and Victor's hand and head positions.  I never observed my ChimeraX (the meeting host) block so the scene must have been sent in a fraction of a second and subsequent command message to Phil also must not have blocked my ChimeraX.

I should measure the bandwidth used sending head and hand positions per person with the default update rate of every frame.

It seems what is needed is the meeting host ChimeraX needs to detect if the messages being sent to the participant are backlogged, and also the participants need to detect if the messages they send to the host are backlogged.  One method would be to not use unlimited buffering.  Then writing the messages would block.  That would freeze ChimeraX unless the message writing were in a separate thread.  The thread could have an input queue and it could drop inessential messages (like hand/head positions) if its input queue grows to span more than a second.  Participant messages to the host could use a similar approach.  I am not sure what performance implications might arise from limiting the socket buffering.  Another approach would be the messages have time stamps.  Participants would also include the stamp of the last message they received and processed in their outgoing messages.  If the host sees that a participant is way behind in processing the messages the participant was sent then it reduces the messages being sent.  Likewise a host message to the participant would identify the last message received by the host from that participant.  If the participant sees that the last message received by the host is old then it can reduce sending outgoing messages.

A related problem is when the network is fast enough to get all the messages but a participant can't process them fast enough.  Currently the code reads every available message before drawing a frame.  If the processing rate is too slow it can end up never drawing a frame or only very infrequently which causes the VR to freeze and black out.  I observed this yesterday using drag model mode and put in a fix where only one motion command message gets processed per frame drawn on the participant's machine, the other available motion command message are dropped."	enhancement	assigned	moderate		VR				phil.cruz@…				all	ChimeraX