Content Comparison

Change History

Overview

This document is intended as a starting point for development of an open synchronized review messaging format in which multiple different applications can participate. The intention is that there is a single presenter and one or more participants with whom the presenter shares his current review session. With the proposed schemas below, the presenter can control playback as well as make annotations on the media and communicate these changes to the participants across a network using a message queuing service. The messages include key exchanges so that they can be encrypted end-to-end to ensure privacy between host and participants.

...

The messages are intended to be used in a presenter-participant style review where there is a single presenter controlling the review session who whose actions will be pushed over a network connection to one or more participants where each user is running their own instance of the application that will reflect the presenter's actions. It is not intended as a collaborative shared session where participants can draw annotations together or any other type of collaborative actions taken. However, if necessary any one of the participants can become the presenter, and doing so will change the current presenter's role to participant.

The details of how the messages are transmitted to the user, and the infrastructure required to do so are not within the scope of this proposal. And as such, no particular message queueing system, database or other requirements are specified. As long as the message arrive arrives in order to the participants, any backend infrastructure should be able to be used.

The messages are also intended to be agnostic of any particular application so that it can be used to share sessions across any application that decides to support the them. However, any application using this system is assumed to support basic review functionality such as media playback and annotations. The messages should also contain mechanisms for embedding application-specific data beyond the scope of synchronizing participants so that applications do not require a separate parallel system to ensure their application-specific features stay in sync.

...

The decision to use OTIO in the messages was made because of several reasons. Among the most important is that it is a rich open source format that many applications already understand, so it seems to be a natural choice to be able to share timelines across both web and desktop applications in order to reduce the burden of adopting the messaging format as much as possible. Another important factor in using OTIO is that a lot of time and effort has already been expended by that team on how to best express and serialize time, a key component of any synchronization. It also provides a rich API around time, eliminating the need to do the (sometimes tricky) mathematical computations around it. Lastly, custom schemas can easily be added to express any effects on the media if need be – like CDLs or other effects for basic luminance, gamma and colour synchronization between apps.

...

An additional design decision was to try to keep the messaging system as lightweight as possible. When a user initially joins a session, they receive a full OTIO Timeline describing the current state of the review. Thereafter, they only receive the changes since their last sync. It's worth nothing noting that the OTIO format does not currently support incremental changes as it was designed as a file storage format more so than a networking protocol. Because of this, schemas that modify a particular clip (like annotations) are always applied to the clip currently being viewed. Exploring partial/incremental updates to OTIO could be a subject worth exploring in the future.

Lastly, the schema system was designed with minimalism in mind. Only messages that directly effect affect synchronization are transmitted with the aim of performing as much of the work as possible on the local system. This includes the actual playback of the media – only media times are sent across the network, no actual pixels are shared as participants are expected to be able to handle playback and access to the media locally.

If the messages are logged to a file, the file format should be JSON Lines format where each message is on a single line. This makes it easy to append to the file, as the review progresses, and it still is a valid file.

Event Data Flow

The below diagrams illustrate the dataflow of the messages between a synced review application, a session store that tracks the current session and message queue for distribution of the messages. The schema for each message is provided in the section following this one.

View file

name	Event Flow.pdf

...

Schema Proposals

The schemas take the form of schema name with a version and a payload. So, in most cases, the json keys come in pairs like "command" and "command_schema" or "event" and "event_schema".

...

Code Block
{ "live_schema": "SYNC_REVIEW_1.0SyncReview.1", "live_payload": { "command_schema": "LIVE_SESSION_1.0LiveSession.1", "command": { "event": "NEW_PRESENTERNewPresenter", "payload": presenter_hash } } }

...

Code Block
{ "live_schema": "SYNC_REVIEW_1.0SyncReview.1", "live_payload": { "command_schema": "LIVE_SESSION_1.0LiveSession.1", "command": { "event": "NEW_PARTICIPANTSNewParticipants", "payload": empty } } }

...

Code Block
{ "live_schema": "SYNC_REVIEW_1.0SyncReview.1", "live_payload": { "command_schema": "SHAREDKEY_SharedKey.1.0", "command": { "event": "GETGet", "payload": public key } } }

...

Shared Key Response

Code Block
{ "live_schema": "SYNC_REVIEW_1.0SyncReview.1", "live_payload": { "command_schema": "SHAREDKEY_SharedKey.1.0", "command": { "event": "SETSet", "payload": hash } } }

Hash is the shared key, encrypted, base64 encoded. It is a symmetric key used between users of the same session, signed with the users public key.

...

Code Block
{ "live_schema": "SYNC_REVIEW_1.0SyncReview.1", "live_payload": { "command_schema": "OTIO_SESSION_1.0OtioSession.1", "command": { "event": "GETGet", "payload": { user: hash, app: id } } } }

...

Code Block

{
    "live_schema": "SYNC_REVIEW_1.0SyncReview.1",
    "live_payload": {
        "command_schema": "OTIO_SESSION_1.0OtioSession.1",
        "command": {
            "events": "SETSet",
            "payload":  {
                  "timestamp": "2025-01-31T16:14:00Z",
                  "otio": OTIO Timeline,
                  "rv"?:
                  "xstudiosg"?: {
                      "xviewscope"?: "full" | "annotations",
           }           "context": }[
    }
}

The otio payload is an OTIO Timeline representing the clip(s) being played. The fields marked with '?' are optional and are meant for sending application-specific information.

Playback Schemas

The next set of schemas are to synchronize playback between users.

Request Sync Playback

Code Block
{ "live_schema": "SYNC_REVIEW_1.0", "live_payload": { "command_schema": "PLAYBACK_SETTINGS_1.0",{ "command": { "event"entity: "GETPlaylist", "payload": empty } } }

The payload is empty here.

Sync Playback

Code Block

language	js

{     "live_schema"id: "SYNC_REVIEW_1.012345",
    "live_payload": {             "command_schema": "PLAYBACK_SETTINGS_1.0",         "command": {},
             "event": "SET",             "payload": {
                "looping"?: true,           /* AND / OR */
   "playing"?: false,                 "muted"?: false,       entity: "Version",
        "playback_range"?: {                    id: "enabled23233": false,
                           }, {
      "zoomed": false,                      entity: "Version",
           "range": {                         "OTIO_SCHEMA"id: "TimeRange.169852",
                        "start_time": {     }
                       "OTIO_SCHEMA": "RationalTime.1"],
                  },
         "value": 0.0,          "otio"? 

                "ratexstudio"?:
30.0                "xview"?:
        },    }
        }
    }
  }

The otio payload is an OTIO Timeline representing the clip(s) being played. The fields marked with '?' are optional and are meant for sending application-specific information.

Playback Schemas

The next set of schemas are to synchronize playback between users.

Request Sync Playback

Code Block

{
    "durationlive_schema": {"SyncReview.1",
    "live_payload": {
           "command_schema": "PlaybackSettings.1",
          "OTIO_SCHEMAcommand": "RationalTime.1",{
              "timestamp": "2025-01-31T16:14:00Z",
             "valueevent": 1.0"Get",
            "payload": empty
        }
     "rate": 30.0,}
}

The payload is empty here.

Sync Playback

Code Block

language	js

{
    "live_schema": "SyncReview.1",
    "live_payload": {
        "command_schema": "PlaybackSettings.1",
        "command": {
            "event": "Set",
         }
    "payload": {
                "timestamp":  }"2025-01-31T16:14:00Z",
                "looping"?: true,
                "playing"?: false,
                "muted"?: false,
                "playback_range"?: {
                    "enabled": false,               
                    "zoomed": false,              
                    "range": {
                        "OTIO_SCHEMA": "TimeRange.1"
                        "start_time": {
                            "OTIO_SCHEMA": "RationalTime.1",
                            "value": 0.0,
                            "rate": 30.0
                        },
                        "duration": {
                            "OTIO_SCHEMA": "RationalTime.1",
                            "value": 1.0,
                            "rate": 30.0,
                        }
                    }
                },
                "current_time"?: {
                    "OTIO_SCHEMA": "RationalTime.1",  
                    "value": 1,
                    "rate": 24
                },
                "scrubbing"?: false,
                "output_bounds"?: {
                    "OTIO_SCHEMA": "Box2d.1",
                    "min": {
                        "OTIO_SCHEMA": "V2d.1",
                        "x": -8.0,
                        "y": -4.5
                    },
                    "max": {
                        "OTIO_SCHEMA": "V2d.1",
                        "x": 8.0,
                        "y": 4.5
                    },
                },
                "source": "/asklfja/3409809".
                "source_index": 0
            }
        }
    }
}

The fields with '?' are optional. The full payload is sent upon first sync and thereafter only the fields that changed since the last sync are sent.

SetCurrentFrame

Set the current frame

Code Block

{
  "live_review_schema": "SyncReview.1",
  "live_review_payload": {
      "command_schema": "PlaybackSettings.1",
      "command": {
        "event": "SetCurrentFrame",
        "payload": {
              "timestamp": "2025-01-31T16:14:00Z",
        	  "start_time": {
                    "OTIO_SCHEMA": "RationalTime.1",
                    "value": 0.0,
                    "rate": 30.0
              }
        }
      }
  }
}

Play
Enable (or disable) Play
{
         “OTIO_SCHEMA”: “play.1”,
         "timestamp": "2025-01-31T16:14:00Z",
         “value”: true,
}


{
"live_review_schema": "SyncReview.1",
"live_review_payload": {
"command_schema": "PlaybackSettings.1",
"command": {
"event": "Play",
"payload": {
                              "timestamp": "2025-01-31T16:14:00Z",
                              “value”: true,
                  }
}
}
}

Loop

Enable (or disable) loop mode.

Code Block

{
    "live_review_schema": "SyncReview.1",
    "live_review_payload": {
          "command_schema": "PlaybackSettings.1",
          "command": {
          "event": "Loop",
          "payload": {
                "timestamp": "2025-01-31T16:14:00Z",
                “value”: true
          }
        }
    }
}

Play

Enable (or disable) Play

Code Block

{
    "live_review_schema": "SyncReview.1",
    "live_review_payload": {
        "command_schema": "PlaybackSettings.1",
        "command": {
              "event": "Play",
              "payload": {
                  "timestamp": "2025-01-31T16:14:00Z",
                  “value”: true
              }
        }
    }
}

Annotation Schemas

The next set of schemas involve user drawn annotations. The event are based on entering the annotation mode, drawing points, ending the annotation mode and clearing annotations.

Coordinate space is X range is -aspect/2 -> aspect/2 y-range = -0.5 -> +0.5

Top-right corner is (aspect/2, +0.5)

Paint Start

Code Block

language	json

{
      },
           "live_schema": "SyncReview.1",
    "currentlive_timepayload"?: {
     
              "OTIO_SCHEMA"command_schema": "RationalTimeAnnotation.1",
        "command": {
            "valueevent": 1,
      "PaintStart",
             "ratepayload": 24{
                  }"timestamp": "2025-01-31T16:14:00Z",
                "scrubbingsource_index"?: false0,
                "output_boundspaint"?: {
                    "OTIO_SCHEMA": "Box2dPaint.1",
                    "minuuid": {
 "foshdfbp5hdirt",
                      "OTIOfriendly_SCHEMAname": "V2d.1Patrick Chevalier",
                        "x"participant_hash": -8.0,
  "d3447b5cb61b41de73a2de39c4f06ab790e66e4cad81f7d449c0147a546244b5",
                     "yrgba": -4.5
                    }[ 1.0, 1.0, 0.0, 1.0 ],
                    "maxtype": {"COLOR",
                        "OTIO_SCHEMA"brush": "V2d.1circle",
                        "xvisible": 8.0true,
                       "name": "yPaint":,
4.5                     },
                },"effect_name": "Paint",
                   "source": "/asklfja/3409809".
layer_range": {
               "source_index": 0        "OTIO_SCHEMA": "TimeRange.1" 
  }         }     }
}

The fields with '?' are optional. The full payload is sent upon first sync and thereafter only the fields that changed since the last sync are sent.

Annotation Schemas

The next set of schemas involve user drawn annotations. The event are based on entering the annotation mode, drawing points, ending the annotation mode and clearing annotations.

Paint Start

Code Block

{        "start_time": {
                            "liveOTIO_schemaSCHEMA": "SYNC_REVIEW_1.0",RationalTime.1",
                  "live_payload": {         "command_schema"value": "ANNOTATION_10.0",
              "command": {             "eventrate": "PAINT_START", 30.0
             "payload": {          },
      "source_index": 0,                 "paintduration": {
                            "OTIO_SCHEMA": "PaintRationalTime.1",
           
        "points": [ ],                     "rgbavalue": [ 1.0, 1.0, 0.0, 1.0 ],
                            "typerate": "COLOR", 30.0
                       "brush": "circle" },
                    "visible": true},
                    "namehold": "Paint"false,
                    "effect_nameghost": "Paint"false,
                    "layerghost_rangebefore": {
   3,
                    "OTIOghost_SCHEMAafter": "TimeRange.1"3
                   }
     "start_time": {       }
        }
    }
}

Paint Point

Code Block

language	none

{
    "OTIOlive_SCHEMAschema": "RationalTimeSyncReview.1",
    "live_payload": {
             "command_schema": "Annotation.1",
        "valuecommand": 0.0,{
            "event": "PaintPoint",
              "ratepayload": 30.0{
                "creation_timestamp": "2025-01-31T16:14:00Z",
      },           "point_target": {
            "duration": {       "source_index": 0,
                    "OTIO_SCHEMAuuid": "RationalTime.1foshdfbp5hdirt",
       
                    "valuerange": 1.0,
   {
                        "rateOTIO_SCHEMA": 30"TimeRange.01"
                        },
"start_time": {
                   },         "OTIO_SCHEMA": "RationalTime.1",
          "hold": false,                     "ghostvalue": false0.0,
                    "ghost_before": 3,       "rate": 30.0
            "ghost_after": 3           },
     }             }      "duration": {
 }     } }

Paint Point

Code Block

{     "live_schema": "SYNC_REVIEW_1.0",     "live_payload": {         "commandOTIO_schemaSCHEMA": "ANNOTATION_1.0",RationalTime.1",
              "command": {             "eventvalue": "PAINT_POINT",
 1.0,
          "payload": {                 "point_targetrate": {30.0,
                    "source_index": 0,   }
                 "range": {  },
                }
     "OTIO_SCHEMA": "TimeRange.1"          "point": {
             "start_time": {        "OTIO_SCHEMA": "Point.1",
                    "OTIO_SCHEMAx": "RationalTime5.1"669550344154889,                
            "value": 0.0,       "y": 2.0246346746398025,
                    "ratesize": 300.005   
                }     },    
            }
       "duration": {}
    }
}

Paint End

Code Block

{
    "live_schema": "SyncReview.1",
    "live_payload": {
         "OTIOcommand_SCHEMAschema": "RationalTimeAnnotation.1",
               "command": {
            "valueevent": 1.0,"PaintEnd",
            "payload": { 
                "rateuuid": 30.0"foshdfbp5hdirt",
                "timestamp": "2025-01-31T16:14:00Z",
		 "points"?: [{
   }                 "OTIO_SCHEMA": "Point.1",
  },                 } "x": 5.669550344154889,               "point": {
                    "OTIO_SCHEMAy": "Point2.1"0246346746398025,
                    "xsize": 5.669550344154889,0.05   
                },{
                    "yOTIO_SCHEMA": 2"Point.02463467463980251",
                    "sizex": 05.05 669550344154889,                
  }                      }"y": 2.0246346746398025,
         }     } }

Paint End

Code Block
{ "live_schemasize": "SYNC_REVIEW_1.0", "live_payload": { 0.05 "command_schema": "ANNOTATION_1.0", "command": { }] "event": "PAINT_END", "payload": { } } } }

Clear

Code Block

{
    "live_schema": "SYNC_REVIEW_1.0SyncReview.1",
    "live_payload": {
        "command_schema": "ANNOTATION_Annotation.1.0",
        "command": { 
            "event": "CLEARClear",
            "payload": {
                "timestamp": "2025-01-31T16:14:00Z",
                "clear_all": false,
                "clear_target"?: {
                    "source_index": 0,
                    "range": {
                        "OTIO_SCHEMA": "TimeRange.1"
                        "start_time": {
                            "OTIO_SCHEMA": "RationalTime.1",
                            "value": 0.0,
                            "rate": 30.0
                        },
                        "duration": {
                            "OTIO_SCHEMA": "RationalTime.1",
                            "value": 1.0,
                            "rate": 30.0,
                        }
                    }
                }     
            } 
        }
    }
}

Version	Old Version 2	New Version Current
Changes made by	Roger Nelson	Sam Richards
Saved on	Jun 30, 2023	Aug 03, 2025

Content Comparison

Versions Compared

Key

Overview

Event Data Flow

Schema Proposals

Shared Key Response

Playback Schemas

Request Sync Playback

Sync Playback

Playback Schemas

Request Sync Playback

Sync Playback

SetCurrentFrame

Loop

Play

Annotation Schemas

Paint Start

Annotation Schemas

Paint Start

Paint Point

Paint Point

Paint End

Paint End

Clear