Vector vs. Raster Annotations Discussion.
- Sam Richards
- Maria Panneer Rajan
This is a summary of a slack discussion, started by Maria Panneer Rajan, with feedback from Dhruv Govil, Nick Porcino, Darby Johnson and Simran Spiller.
Storing Annotated Drawings as Raster-Images-with-Alpha
For 2D media ( image-sequences/quicktimes ) reviews, what is the value we are providing to artists and reviewers by storing the reviewer's annotated drawings as a vector ?Can't we just render the annotated drawings created by a reviewer on respective frames into raster-images-with-alpha using a format such as PNG ?And then, when the artist or reviewer wants to see that annotation that was done on a particular version of daily media ( image-sequence/quicktime ) we can just overlay the appropriate annotated raster-image-with-alpha on top right ?
Q: @Sam Richards : Does the alpha contain the brushstrokes for the annotations rather than the original image alpha? Note, I really like the idea that you could take the annotations themselves and comp them over other versions of the image, as well as having the original image with it. The downside is that it may force the image to be a png rather than a more compact jpg.
A: @Maria Panneer Rajan : Yes Sam, it will be good to create a raster image with the alpha of only the brush strokes.
For Example,
Lets say the following is the original daily that a Reviewer sees in her preferred player's viewport,
Now on top of the original daily, using the vector-annotation-tools available in the player that the Reviewer is using, she can create her vector annotations like so,
After the Reviewer has completed drawing all the vector-annotations on-top of the original daily, she will be publishing the vector-annotations to the artist.
At this point, we convert only the vector-annotations that were drawn on-top of the original daily into a raster-image-with-alpha as show in the following image,
In this way, we can overlay this annotated-raster-image-with-alpha on-top of the original daily anytime we want to see the annotation that was created by the Reviewer.
Advantages of using Annotated-Raster-Images-with-Alpha
Since these annotated-raster-images-with-alpha will be rendered with the same resolution as the daily ( image-sequence/quicktime ) on top of which they were drawn on, the annotated-raster-images-with-alpha will always preserve the same quality as the daily ( image-sequence/quicktime ) no matter what device they are played on.
Even when the resolution of the daily (image-sequence/quicktime ) needs to changed to create a down-rezed proxy, the same down-rezing can also be done to the annotated-raster-images-with-alpha.
The same is true even when you need to crop the daily (image-sequence/quicktime ), you can do the same cropping to the corresponding annotated-raster-images-with-alpha .
And since we are storing only the annotated-drawings ( and not redundantly storing the image below ) as raster-images-with-alpha the file size of these will also be small.
And since we are literally creating a rendered annotated-raster-image-with-alpha, there is no limitation to the drawing tools ( brushes, strokes, symbols, stickers, emojis, etc. ) that a reviewer can use while doing their reviews.
Reproducibility is a bit of a concern (missing fonts, different text or emoji rendering, etc.), whereas with raster images you see exactly what the annotator saw, even though you may not be able to re-create it in your tool / on your system.
Advantages in using Vector Annotations
At-least based on my limited understanding, for 2D reviews, the only advantage of storing annotations as vectors is, after importing back a vector-annotation we will be able to edit/modify the annotated vector strokes more easily. But 90% of the time, reviewers/artists won't be editing/modifying already drawn annotations right ? - Sam: The assumption is that we are not editing them.
For reviewing in a 3D space storing the annotated strokes as vectors will be beneficial.
Vectors are much smaller data for the average annotation. They therefore transfer more easily in sync sessions. For more complex annotations a raster image might become smaller.
Vectors map better to the recorded input so it's easier to develop with, because you don't have to manage undo stacks with different data
Easier to append metadata like user id to strokes with vectors for multi session recordings, also useful to timestamp strokes to align them to audio/speech to text.
For the case where text is part of the annotation, the text can be searchable.
Much easier to have a single file containing summary of a review, including all notes and annotations.
It would be possible to render lines using a constant stroke width regardless of the zoom factor, to have more pixels of the footage visible
Color could be adjusted dynamically (or an outline or drop shadow added) to improve contrast, potentially helping with accessibility
In the rare event of the aspect ratio being changed, vectors can be stretched without introducing artifacts. However, a more common scenario would be that an annotation is (accidentally) created for a low res proxy, and thanks to resolution independence it can be scaled to full resolution without artifact
Its easy to have the annotations go outside of the image, some users have used it as a canvas for drawing alternatives or writing comments on.
Additional Comments
The original proposal is really to focus on how do we make vector annotations easy to use through the system, from providing the metadata in the raw media to define the coordinate system. Maria is correct that writing a translator that could correctly interpret them can be challenging, which is why we think its a significant project. But its also worth considering whether a hybrid vector/image system in terms of actual interchange might give a decent fallback if the vectors dont work.
Image-Formats worth exploring for creating Annotated-Raster-Images-With-Alpha:
Darby Johnston:
That's a good point about the raster images being more portable; while drawing lines and rectangles isn't very difficult, when you inevitably want to add text/fonts/emojis it gets a lot more complicated. You could probably get pretty small files sizes with PNG, but the decompression time could have a big impact on playback performance...
Nick Porcino:
turbojpeg holds up with suitable settings. I use it in realtime applications all the time, like for streaming camera feeds. As engineers we might want lossless, but in practice, if an annotation like "MOAH COWBELL ----> " has got a tiny bit of compression artifacting, it might be ok, just saying
Simran Spiller:
How would OpenEXR B44 compare in terms of performance? It has a fast compression scheme and values would already be floating point numbers, which may or may not be an advantage for comping. It's not a good choice in terms of file size, however.
In our Slack channel, Darby Johnston brought up an important point of how the decompression time of PNG might hit the playback performance of the dailies when the PNG is overlaid on top of the daily.
Hence the following image formats where suggested by Nick Porcino and Simram Spiller for exploring as an alternative to PNG,
turbojpeg
OpenEXR B44