This is a pain in the butt for me. Im working with projects that have a lot of effects and optical flow data. Every time I switch from project to project or FCPX crashes or I reboot en re-open the Library the renders are all messed up. Some parts are renderen some aren't. and then to scrub through the footage I have to re-render the project all over again. The data (so says Final Cut Library Manager) in tact, size wise that is. This is costing me hours of pre-rendering!
I'm going to get a new Mac Pro soon with an afterburner, so the time will decrease a lot but it's still VERY annoying.
Xavier Novembre wrote: the strange thing is that it seems to happen a lot to some people and never to others (like me)
- Add a short ProRes clip to a new timeline
- Add the effect "Aged Film"
- Render timeline with CTRL+R
- Note render dots go away
- Restart FCPX and see if it's still rendered (it probably won't be)
But the problem is much worse than this. After the initial render, no render dots are shown, yet FCPX will not use the render files. Exporting that PR422 timeline to a PR422 output will be very slow.
If you then relaunch FCPX, the dots indicate the render files were either lost or considered invalid. If you THEN render the timeline, THEN export to PR422, it will be about 5x faster.
Apparently there are many conditions under which timeline ranges show as rendered but FCPX will not use those render files. This may account for various unexplained slowdowns during export to ProRes 422.
I will be doing further testing today and will contact ProApps support.
Further testing shows there are multiple cases, each with unique behavior. All of the below use 4k PR422 source, a fully-rendered timeline, and 4k PR422 output. Most testing was on a 10-core Vega 64 iMac Pro, FCPX 10.4.8 and Mojave 10.14.6. Limited testing on a 2017 iMac 27 running the same versions and a i9 MBP 16 running FCPX 10.4.8 and Catalina 10.15.2 did not show any difference. Likewise, limited testing on FCPX 10.4.6 and 10.4.7 showed no difference. This behavior has been here since 10.4.6; I don't know how long prior to that.
Case 1: Individual Fx can be rendered in timeline and render files are used for PR422 export. IOW the Fx is not re-computed. Export is fast after rendering because render files are used. Typical export speed: 7 sec for a rendered 60 sec 4k timeline. Many FCPX built-in Fx are in this category.
Case 2: Some Fx show as rendered but the render files are not used for export to PR422. Re-launching FCPX makes the timeline range containing the Fx show as non-rendered. Re-rendering it then enables those render files for export -- IOW export is about 4x or 5x faster if I/O subsystem permits, 7 sec for a 60 sec timeline on an iMac Pro. Examples: Aged Film, Aged Paper, Cross Hatch, Film Grain, Textures, Water Pane.
Case 3: Some Fx are like Case 2, except they are permanently slow. Even after re-launching FCPX and re-rendering the timeline, export to PR422 will always be slow. Typical: 35 sec for a 60 sec timeline, about 5x slower than the cached case. Apparently the render files are not being used, even though no render dots are shown. Examples: Artifacts, Bokeh Random, Camcorder. Upon re-launch of FCPX the render dots will sometimes briefly flicker on then vanish. Note background rendering is off.
Case 4: Mixing certain "OK" Fx from Case 1 may produce a non-cacheable render state. As a group they become like Case 3, non-cacheable so export to PR422 will always be slow. No render dots but cache isn't being used for PR422 export.
Case 5: Mixing certain Fx from Case 2 which are *initially* non-cachable when used singly then as a group become permanently non-cacheable, ie Case 3.
Another finding: Using the "poorly behaved" (from a render cache standpoint) built-in Fx in conjunction with Neat Video will discard render files upon FCPX re-start, or else may result in a non-usable render cache.
By contrast Neat Video plus a "well behaved" Fx will preserve the render files upon FCPX re-start.
OTOH using Neat Video with Aged Paper will result in all render files in that range being invalidated upon FCPX re-start.
Using Neat Video with Water Pane will result in non-usable render files. IOW no matter how many times you render the timeline, even though no render dots exist, during export to ProRes 422 it will never use the render files, thus will recompute Neat Video upon each export.
There is a crazy quilt of combinations and behaviors, but the bottom line is render files are not being properly cached and re-used. The issue is not "why would you ever use Neat Video on the same clip as Aged Paper?" Rather this illustrates the render cache system is malfunctioning, probably under many conditions. This is just the tip of the iceberg.
We have known for some time the render cache doesn't always work right but it wasn't clear until now the extreme cost this can have.
I had several long conversations with Apple Pro Apps support about this the previous two days.. Unfortunately neither they or Engineering admit it's a problem. In fact both Pro Apps support and Engineering categorically state that render files are never used for exporting, therefore any alleged impact on export performance is a non issue.
This is obviously not correct, and can be easily reproduced. There are two different underlying methods by which render files can accelerate export:
(1) Any compute-intensive effect is pre-rendered so this will not normally require re-computing during export. E.g, if you pre-render a timeline containing Neat Video, Imagenomic Portraiture or Digital Anarchy Flicker Free, each subsequent export may be 50x faster. If FCPX mis-manages the cache and doesn't use it for export, the full compute burden of the effect will be paid for every export.
(2) If the export codec is ProRes 422, a special "fast path" export exists. In this case FCPX will simply concatenate the individual render segments and write them to the output file, with no encoding. On my iMac Pro writing to a SSD array, this produces about 700 megabytes/sec export rates. In this case you save both effect rendering plus any encode cost.
This can be easily demonstrated, although for the ProRes case it requires a fast SSD to fully appreciate the difference.
- Import a 60 sec 4k ProRes 422 clip
- Add Color Wheel effect and make any adjustment. Render dots appear.
- Do timed export to 4k ProRes 422 on a fast SSD. On my 10-core Vega 64 iMac Pro, writing to a 4-drive Thunderbolt SSD array, this takes about 22 sec
- Render using CTRL+R. Render dots vanish.
- Repeat above export. On my machine the export time diminishes from 22 sec to 6.9 sec, or about 3x faster. This is 100% reproducible. The more compute-intensive the effect, the greater the *export* performance benefit of pre-rendering the timeline.
Certain "poorly behaved" built-in effects if used in the same stack with Neat Video or other 3rd-party compute-intensive effects will prevent render files for those clips from being used during export. An easy example is add Neat Video to a clip, build a NR profile, render then export to ProRes 422. It will export in seconds because the compute-intensive Neat Video is already processed to render files. Then add the built-in effect "Aged Paper", then re-render, then export. Export will be permanently slow, even though no render dots are showing. The built-in effect is somehow preventing use of render files for export, thus each export with Neat Video will be very slow.
The problematic built-in effects I've isolated so far are: Aged Film, Aged Paper, Artifacts, Bokeh Random, Camcorder, Cross Hatch, Film Grain, Textures and Water Pane. The other built-in effects seem to work OK.
The issue is not "why would you ever use Aged Film on a clip with Neat Video?" It just illustrates the problem. The actual mis-management of render files is likely broader than a few built-in effects. In limited testing it seems certain Motion templates also cause this result. It's conceivable that many different things cause this, e.g, Titles, adjustment layers, etc.
To my knowledge, effects can be constructed three different ways: (1) As a wrapper for a color correction effect, (2) As a Motion template, and (3) As Objective-C code built using the FxPlug SDK. It's possible the render cache behavior varies with each type, or maybe certain quirks about Motion-implemented effects prevent use of render cache for export.