Artificial Intelligence and Machine Learning are changing Post Production processes for the better, with tools and techniques that are being developed to leverage these rapidly improving technological advances delivering almost magical results.
In this article, we’ll take a quick tour of three of the most exciting AI and ML driven tools you can use today, and a couple that are already living inside of your video editing software of choice.
The Difference Between AI and ML
Before we get to the fun stuff, it’s worth a very brief pause to quickly explain that we are quite some way from an I, Robot or Terminator future.
“Artificial intelligence is the science and engineering of making computers behave in ways that, until recently, we thought required human intelligence.”
- Andrew Moore, Carnegie Mellon University
When most people think of AI, they think of intelligent machines that can think and act upon their own objectives.
Yet, as far as I’m aware, almost every mention of ‘AI’ in the marketing of these post production tools is actually talking about Machine Learning (ML), which is quite a different thing.
ML is a branch of AI and far more constrained in its scope and ability:
Given a large enough dataset of images (e.g. people smiling), the computer program would be able to iteratively learn what a smile looks like, based on patterns of pixels, and so eventually, predictively and accurately find and tag photos of people smiling.
“Machine learning is the study of computer algorithms that allow computer programs to automatically improve through experience.”
- Tom M. Mitchell, Computer Scientist
This same approach allows the 'Neural Engine' in DaVinci Resolve Studio to power features such as "facial recognition, object detection, smart reframing, speed warp retiming, super scale upscaling, auto color, color matching and more.”
Similarly, Adobe’s Sensei powers a host of different features across the Creative Cloud suite of applications including content-aware-fill in After Effects and Auto-ducking, Morph Cut, Color Match and Auto Reframe in Premiere Pro.
You can read more about Adobe’s ‘AI’ powered creativity here. What about other emerging post production tools using AI?
Tool No. 1 - No More Rotoscoping
The problem with traditional rotoscoping techniques is that they are a manual and painstakingly time consuming process. For example, drawing a cutout around a moving shape frame by frame. That’s where AI based tools that have been trained to recognize the edges of shapes can save hundreds of hours by doing the heavy lifting for you.
Runway is a browser-based video editing tool which offers a really precise ‘green-screen’ tool (AI powered rotoscoping) but also includes some other really useful AI based tools, such as:
- Green Screen - automatically remove backgrounds/rotoscope foregrounds
- Inpainting - Remove objects from a shot with smart ‘in-painting’
- Automatic Beat Detection - Pre-edit to the beat
- Automatic Subtitles
- Automatic Background Noise Removal
They also have an interesting platform called Runway Model Playground through which they share their growing catalog of machine learning models which includes depth estimation, colorizing black and white images and object detection.
Runway seems to be one of the main companies who are pushing the boundaries of what is possible in an online video editor that you can use for free, and are definitely ones to watch.
Tool No. 2 - Color Grading in the Cloud
Making every shot in a final sequence have the same look and feel regardless of how varied their starting points is one of the reasons professional colorists exist. It takes real skill and a practiced eye to make everything look “normal”. It also tends to be the less creative part of the process compared to the stylistic look creation side of things.
Here again AI based tools can offer great results by analyzing different sets of images and adjusting the levels of each to make them all more closely align.
fylm.ai is a professional grade, in-browser color grading application equipped with a very rich set of tools including the ability to:
- Replicate the look of any reference image and apply it to your footage
- Match the sensor characteristics of any two cameras to each other
- Re-create what a LUT is doing based on matching a before and after image
This automatic shot matching capability helps to ensure consistent grades across your project as you match the look of your hero shot to all your other shots in a scene. This is often easier said than done for beginners when relying on your eyes and color grading scopes.
Fylm.ai recently added two impressive new tools to their arsenal including AI Auto Correct and AI Color Match. All of fylm.ai’s tools have also been improved with the option to “favor context” or “favor absoluteness” to give you more sensible control over what the algorithm applies, depending on your intentions.
One of the huge benefits of flym.ai is that it can work with any raw camera file, as opposed to having to upload a low-res proxy.
If you’re a professional colorist, or an editor who needs all the help you can get, having tools like fylm.ai up your sleeve could be a huge timesaver.
Tool No. 3 - Logging, Tagging and More the Easy Way
One of the traditional tasks for an assistant editor is to do much of the “grunt work” of ingesting, organizing, categorizing, labeling and generally preparing the footage ready for the creative editorial phase.
This usually takes a long time and a lot of repetitive actions, both of which can be eliminated thanks to machine learning techniques for recognizing objects and adding the associated metadata.
Nova.ai has a huge range of AI driven post production tools that can do everything from visual searches of your footage to automatic translation into 30 other languages using transcription and subtitle translation.
As they have so many tools on offer, I’ll focus on one of the most interesting:
The ability to search your video to find and tag a range of facial emotions, objects and activities (coat, car, tennis, eating) as well as celebrity faces.
You can also upload an image of a person and then get Nova to search your footage for that person.
This could make traditionally time consuming editorial tasks much faster such as finding all of the close ups of a particular actor for a trailer or all of the goals in a football match for a highlights reel.
The days of trawling through archives for shots of particular people or places would be a thing of the past.
The only catch to all this magic is that you will need to upload and store everything you want to be able to search to their cloud storage platform as all of the storage and processing takes place on their servers.
There is more to come
These three tools are just the tip of the iceberg when it comes to how artificial intelligence and machine learning are changing and will continue to change post production workflows.
Other new AI/ML driven technologies such as the ability to generate synthetic voices are compelling as well. These kinds of text-to-speech tools make creating and re-creating temp voiceovers extremely easy.
And while many of these kinds of tools will eventually make their way into desktop applications like DaVinci Resolve, After Effects or Premiere Pro (where you’ll be able to put these powerful algorithms to work directly in your timeline), knowing today what can be achieved in your browser is a super-power in itself.
With integrations into Adobe Premiere Pro and other Creative Cloud apps, MediaSilo allows for easy management of your media files, seamless collaboration for critical feedback and out of the box synchronization with your timeline for efficient changes. See how MediaSilo is powering modern post production workflows with a 14-day free trial.
