Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1
  • 2

TOPIC: Render file question prior to Masterfile

Render file question prior to Masterfile 01 Aug 2018 17:31 #96630

  • Jonathan Levin
  • Jonathan Levin's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 679
  • Thank you received: 17
  • Karma: 5
Hi all.

After a daily edit session, I generally do some some house cleaning. I select a library, and then delete all unused render files, mostly out of concern of Library becoming to big.

After final edit and picture lock is it good practice to select library then delete ALL render files, and then got to Project, Re-render desired project(s) and then export master file (or whatever flavor)???

My gut feeling tells me this is the right way to do this. Thoughts?
The administrator has disabled public write access.

Render file question prior to Masterfile 02 Aug 2018 00:12 #96634

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2185
  • Thank you received: 294
  • Karma: 33
It's always good to delete all render files, yes. But, why render a timeline before exporting? Seems a waste of time and disk space.
The administrator has disabled public write access.

Render file question prior to Masterfile 02 Aug 2018 06:24 #96641

  • Axel
  • Axel's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 397
  • Thank you received: 51
  • Karma: 9
Depends on your machine. I render selected when playback isn't perfectly smooth (retiming with optical flow) or when there is no realtime at all (Neat). On my 2015 iMac, export times (to a ProRes Master) doesn't take longer without having rendered all in advance, at least not significantly. Not that I mind. With Library Manager, I can see the ratio between Original, Flow and Renders displayed. If render files take up less than 20% of all (which is typical for my stuff), I don't erase them. That way, when a client (in my case: the newly weds) comes to review the edit, playback is still smooth.

For shere playback, it doesn't seem to matter if you use UHD long-GOP original media, Optimized or Proxy. That is: with Quicksync. However, I noticed that with Original Media only, "j" and "<" are not frame-accurate. Can anyone confirm? So for me the way to go is using Proxy for editing and changing to Original for CC.

Playback of Proxy with the a.m. effects applied isn't any better than with Original.
Last Edit: 02 Aug 2018 06:31 by Axel.
The administrator has disabled public write access.

Render file question prior to Masterfile 02 Aug 2018 11:20 #96646

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2185
  • Thank you received: 294
  • Karma: 33
Yes, for playback, rendering helps, but the OP asks about, or it seems to me, rendering before (for the purpose of) exporting.

JKL and < > keys all work fine for me with original media. We use nothing but, never use proxies, and often have media from 2-4 different cameras. What exactly are you seeing when those commands are not accurate?
The administrator has disabled public write access.

Render file question prior to Masterfile 02 Aug 2018 15:18 #96648

  • Jonathan Levin
  • Jonathan Levin's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 679
  • Thank you received: 17
  • Karma: 5
Thanks guys.

The reason I ask about rendering before export is that I seem to recall it being recommended in the FCP 6 and 7 days. I could be recalling incorrectly though. For archiving I delete ALL render files and if need to revisit a project later on I render it out.

Axel- I render out sections of a project where I want to see how things are looking with a smooth playback, like when I do some sort of title build effect thing, or adding effects to clips in project. I sometimes try this using the Better Quality setting in the viewer as well, then switch back to better Performance..

IIRC the downside of not rendering prior to export is that the export takes longer to complete. Again, going from the old FCP 7 and 6 days. Is it bad practice to render prior to export or just not necessary?
The administrator has disabled public write access.

Render file question prior to Masterfile 02 Aug 2018 21:34 #96653

  • Axel
  • Axel's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 397
  • Thank you received: 51
  • Karma: 9
FCPX.guru wrote:
JKL and < > keys all work fine for me with original media. We use nothing but, never use proxies, and often have media from 2-4 different cameras. What exactly are you seeing when those commands are not accurate?


Well, actually, most of the time it works fine. I don't know why, but sometimes reverse playback pauses briefly at the clip boarders and then skips a few frames to catch up. Same with one frame backwards ("<"), hit repeatedly. These I consider minor glitches. It gets annoying when it happens in detailed trimming feedback when using the comma. Weird, I can't reproduce it now ...

Jonathan Levin wrote:
IIRC the downside of not rendering prior to export is that the export takes longer to complete. Again, going from the old FCP 7 and 6 days. Is it bad practice to render prior to export or just not necessary?


Like I said before, I think it depends on whether your Mac has Quicksync or not. Try it. Of course, with a lot of effects applied, exporting to a ProRes master with a pre-rendered timeline (given that ProRes is also your render codec) *should* just mean copying everything to one self-contained file (if FCP X behaves like FCP 7 in this respect, which I don't know) and will be faster. But - compared to, say, a 2009 cheese grater with FCP 7 - the difference can be negligible. Export the rendered project and stop the time. Then delete all render files and export again.
Last Edit: 02 Aug 2018 21:37 by Axel.
The administrator has disabled public write access.

Render file question prior to Masterfile 02 Aug 2018 23:20 #96654

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2185
  • Thank you received: 294
  • Karma: 33
OK, this was true in 6 & 7, and is now for X. IF you render your timeline first, it will export faster. Was more so for 6 & 7. X exports faster than they did no matter what.

BUT, there is a catch. If you INCLUDE the time you took to render the timeline, PLUS the time it took to export, you spent more time than you would have if you simply exported with no rendering at all in the first place.

I worked with a publisher on testing this when X came out. We ran tests on legacy and X (10.2 I think). We revisited the tests in 10.3. Still holds up today.
The administrator has disabled public write access.

Render file question prior to Masterfile 03 Aug 2018 00:55 #96660

  • Tom Wolsky
  • Tom Wolsky's Avatar
  • OFFLINE
  • Moderator
  • Posts: 4128
  • Thank you received: 662
  • Karma: 106
There is some speed improvement if you’re going to PR422 or whatever your render spec is. There isn’t if you’re going to H.264, and certainly not if you have the hardware codec.
The administrator has disabled public write access.

Render file question prior to Masterfile 03 Aug 2018 01:14 #96661

  • Jonathan Levin
  • Jonathan Levin's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 679
  • Thank you received: 17
  • Karma: 5
Thanks again!

Real interesting stuff.

Most of the video I edit is PR422 from Atomos external recorder. And for Masterfile/Archive purpose I also go with PR422. From that file I’ll use Compressor to transcode for YouTube, IPad, whatever.

Of coarse when you start adding titles, effects, the occssional POS H.264 clip, rendering does help.

Someday I suppose rendering and render files will be a thing of the past, like interlaced video. Wouldn’t that be cool B)
The administrator has disabled public write access.

Render file question prior to Masterfile 03 Aug 2018 06:06 #96665

  • Axel
  • Axel's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 397
  • Thank you received: 51
  • Karma: 9
Jonathan Levin wrote:
Most of the video I edit is PR422 from Atomos external recorder. And for Masterfile/Archive purpose I also go with PR422. From that file I’ll use Compressor to transcode for YouTube, IPad, whatever.

We've come a long way. Remember when there were races between FCP 6/7 and Premiere CS, where FCP regularly lost because mpeg4 needed the log&transfer procedure? We argued that on the long run a meticulously logged project was an advantage. True, as I see it. The Premiere guys said that only the native codec provided the best quality - but there was never a difference in quality (because ProRes was so good). With FCP X, we have the best of both worlds. We are able to edit the native codec and have our Library neatly organized in no time (but I don't count that, because I find organizing footage actually enjoyable).

Back then - only ten years or so from now - editing highly compressed HD presented a challenge for older or weaker machines. Today I record, edit and screen my UHD vacation videos on my iPhone. You take some care and use Filmic Pro and a $150 smartphone gimbal, and it holds up well on a big display/TV ...

The codec implementations also improved. AVCHD only 10 years ago was abysmal. The naked eye could spot compression artifacts. Today pixel peepers compare externally recorded ProResHQ to the camera file and have a hard time justifying the additional hassle - it usually only pays off if it's true 10-bit that the Atomos delivers. Saw a clip on Youtube, were Max Yuryev compared internal 8-bit SLog2 from FS5 with Atomos ProRes Raw : the threadbare camera recording held up *better* in post!

To make a long story short, we older guys who experienced all the limitations of the past are accustomed to procedures that are 'good' but not always necessary. I will never export directly to H.264/265. After having spent many hours to edit and polish the video, I don't want to export directly to a highly compressed upload file. My way more professional Premiere buddy mocks my stubbornness 'cause that's what he did for years - with no visible differences. Since a year or so he also occasionally edits with FCP X, and he says, why, you don't need these big ProRes files at all!
Last Edit: 03 Aug 2018 06:20 by Axel.
The administrator has disabled public write access.

Render file question prior to Masterfile 03 Aug 2018 09:53 #96666

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1009
  • Thank you received: 211
  • Karma: 17
Tom Wolsky wrote:
There is some speed improvement if you’re going to PR422 or whatever your render spec is. There isn’t if you’re going to H.264, and certainly not if you have the hardware codec.

This is an important point. ON H.264, the encoding overhead can smother the performance advantage of a pre-rendered timeline. This is so dramatic it can appear that FCPX doesn't even use existing render files. E.g, you can add many built-in effects, yet pre-rendering might not make much difference for H.264 export performance.

By contrast with a PR422 source and PR422 export, a pre-rendered timeline makes a bigger difference. The encoding phase is easier thus the rendering phase becomes the limiting factor.

However even on H264 a pre-rendered timeline will help for extremely compute-intensive effects such as Neat Video, Imagenomic Portraiture, Digital Anarchy Flicker Free, etc.

If render files don't exist, the export process must render each clip range before it can be encoded. But those render files are not saved. Thus there are cases where it could be advantageous to manually render the timeline.

E.g, you are nearing completion of a complex project and have used many compute-intensive effects. Yet the timeline will probably be tweaked and exported 20 more times for various review passes. If it's not pre-rendered, each export could be quite lengthy due to constantly re-rendering the effects. If you pre-render the timeline just once, then each subsequent export will be a lot faster -- possibly even on H.264.
The administrator has disabled public write access.

Render file question prior to Masterfile 03 Aug 2018 10:06 #96667

  • Axel
  • Axel's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 397
  • Thank you received: 51
  • Karma: 9
Audio work can take a while. If you make that the last step, - that is: when your picture is locked, - you could turn on background rendering. That way the timeline could be rendered before you finished cleaning up audio.
The administrator has disabled public write access.

Render file question prior to Masterfile 03 Aug 2018 11:00 #96672

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2185
  • Thank you received: 294
  • Karma: 33
Again, our real-word tests showed pre-rending a timeline only gives a very slight increase in export times, no matter what codec, and the time spend pre-rending makes it an even longer process. Background rendering? You're talking about saving 3 seconds, not minutes.
The administrator has disabled public write access.

Render file question prior to Masterfile 03 Aug 2018 15:56 #96677

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1009
  • Thank you received: 211
  • Karma: 17
FCPX.guru wrote:
Again, our real-word tests showed pre-rending a timeline only gives a very slight increase in export times, no matter what codec, and the time spend pre-rending makes it an even longer process. Background rendering? You're talking about saving 3 seconds, not minutes.

In general I agree that there's sometimes little advantage. Because of this discussion I did some simple tests that indicated no benefit for H264 output, even if using several built-in effects. I also did some simple tests that showed more benefit to pre-rendering for ProRes source and ProRes output -- thus my previous comments.

However I then loaded a short but very complex real-world timeline from a documentary I just finished, and did numerous export tests to various codecs, with and without a pre-rendered timeline. The results showed wide but consistent variation -- see below.

Project details: 4 min 30 sec documentary using UHD 4k H.264 100 mbps 8-bit 4:2:0 source. Much of this was multicam and the timeline contained thousands of edits and hundreds of effect applications, inc'l lots of keyframing. Most of the effects were built-in, but three were 3rd party: CoreMelt's SliceX, Neat Video, and Digital Anarchy's Flicker Free.

I first deleted all the render files and exported it to Pro Res 422. I then pre-rendered the timeline and repeated the test. I then rebooted, deleted the render files and repeated the test to check consistency. I then repeated that again using 1080p and UHD 4k exports using both "Fast" and "Better Quality" modes. The Fast mode will use Quick Sync or AMD's VCE if available, whereas the Better Quality mode does not.

System config: 10-core Vega 64 iMac Pro, 64GB RAM, 2TB SSD, macOS 10.13.6, FCPX 10.4.3. Media and library files on OWC Thunderbay 4 Mini using 4 x 2TB SSD drives in RAID-0. Output file goes to iMac Pro 2TB SSD.

Export to 4k ProRes 422 with no existing render files: 9 min 31 sec
Export to 4k ProRes 422 with fully pre-rendered timeline: 9 min 25 sec
Timeline pre-render time (all cases, tested several times): 9 min 28 sec

I then rebooted, deleted all render files and repeated the test but exporting to 1080p and 4k H.264, using both fast and "better quality" encodes:

Export to 1080p H.264 Fast with no existing render files: 9 min 56 sec
Export to 1080p H.264 Fast with fully pre-rendered timeline: 1 min 45 sec

Export to 1080p H.264 Better Quality with no existing render files: 19 min 41 sec
Export to 1080p H.264 Better Quality with fully pre-rendered timeline: 3 min 22 sec

Export to 4k H.264 Fast with no existing render files: 10 min 13 sec
Export to 4k H.264 Fast with fully pre-rendered timeline: 3 min 14 sec

Export to 4k H.264 Better Quality with no existing render files: 21 min 15 sec
Export to 4k H.264 Better Quality with fully pre-rendered timeline: 6 min 25 sec

For this particular project, when encoding to ProRes 422, pre-rendering the timeline costs a lot, yet contributes almost nothing -- despite encoding to PR422 which should lessen the encode burden. This is the opposite of what I saw on some simpler tests.

Encoding to H.264 showed wide (but consistent) variation, depending on whether the output used "Fast" or "Better Quality" encoding. In general there was significant benefit to pre-rendering the timeline.

There are definitely cases where it doesn't help to pre-render the timeline, and the total time of pre-rendering plus exporting can be greater than just exporting it without pre-rendering. However there are other cases where it makes a huge difference. It is difficult to predict, and I can't explain why there is so much variation.

I don't have time to re-run all these on my 2017 iMac but that might show additional differences since the # of cores, GPU and Quick Sync vs AMD's VCE might skew the relative proportion of render time to encode time.
Last Edit: 03 Aug 2018 16:00 by joema.
The administrator has disabled public write access.

Render file question prior to Masterfile 03 Aug 2018 22:19 #96694

  • Jonathan Levin
  • Jonathan Levin's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 679
  • Thank you received: 17
  • Karma: 5
You guys are insane. But thats why I love you!
The administrator has disabled public write access.

Render file question prior to Masterfile 04 Aug 2018 01:21 #96696

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2185
  • Thank you received: 294
  • Karma: 33
How much time did it take to render that timeline each time? Add that to your export time for a total. That was my original point. How much faster is it, with timeline render time plus export time together?
Last Edit: 04 Aug 2018 01:23 by FCPX.guru.
The administrator has disabled public write access.

Render file question prior to Masterfile 04 Aug 2018 01:36 #96698

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1009
  • Thank you received: 211
  • Karma: 17
FCPX.guru wrote:
How much time did it take to render that timeline each time? Add that to your export time for a total. That was my original point. How much faster is it, with timeline render time plus export time together?

Each time the timeline render was about 9 min 28 sec. However you generally only have to render it once, after that the render files are still there for any range which hasn't been invalidated by a subsequent change. This becomes a factor with a complex timeline which is approaching completion (and a delivery deadline), yet which is undergoing minor changes yet is repeatedly being exported for final evaluation. The final changes are only affecting small regions of the timeline, so only those small portions have invalidated render files. Most of the previously-built render files are still used to accelerate the final export.

In the the late stages of development there can be thousands of cumulative edits in the timeline and all the compute-intensive effects are there. The increase in total export time is huge vs the same-length original assembly when only 20% of the edits are there and the compute-intensive effects (stabilization, de-flicker, skin processing, object removal) haven't been added.

In the late stages, pre-rendering the timeline seems to help a lot for subsequent exports, at least for H.264 encoding. ProRes output is different.
Last Edit: 04 Aug 2018 01:37 by joema.
The administrator has disabled public write access.

Render file question prior to Masterfile 04 Aug 2018 01:40 #96699

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2185
  • Thank you received: 294
  • Karma: 33
So that "Export to 4k H.264 Fast with fully pre-rendered timeline: 3 min 14 sec" is actually 12:42 total time to achieve the export. Compared to "Export to 4k H.264 Fast with no existing render files: 10 min 13 sec" which is a bit shorter. The article we published included timeline render times, as you can't get the faster export without it. Just saying. Interesting topic.
The administrator has disabled public write access.

Render file question prior to Masterfile 04 Aug 2018 06:06 #96702

…actually 12:42 total time to achieve the export. Compared to … 10 min 13 sec…Interesting topic.

<cough> … gentlemen, with all due respect ;)
a 4:30min project needs 150 sec more or less, one way exported or another …
Interesting in the NewsRoom, and probably, when client Mr Superimportant stands behind you, starring on his Rolex, "I do have my next appointment now!"

… in real life, get another mug of coffee, leave chair for stretching, check your tweets, … even keep on editing the Project (one of the many marvels of FCPX, imho …), no need to stare the cake piece grow... (hm? get a piece of cake, another option to spend 150secs …)

Technically indeed interesting … but not affecting my life (creeping under my stone)
Just an opinion!

:)
The administrator has disabled public write access.

Render file question prior to Masterfile 04 Aug 2018 10:57 #96705

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1009
  • Thank you received: 211
  • Karma: 17
FCPX.guru wrote:
So that "Export to 4k H.264 Fast with fully pre-rendered timeline: 3 min 14 sec" is actually 12:42 total time to achieve the export. we published included timeline render times, as you can't get the faster export without it....

As I stated, in all my test cases the timeline render time was 9 min 28 sec. However on some common workflows you only pay this one time. You need not re-render before every export, because most of the timeline is still rendered.

If your workflow is you never export until the product is totally done, then you export one time and that one export is 100% perfect, passes all reviews and is published -- in that case pre-rendering the timeline won't help.

But if you must do multiple exports amid the last few changes -- especially at the later phases of development when the timeline may be burdened with many edits and effects -- in that case pre-rendering can help considerably. You only pre-render one time, then might do 10 or 15 exports, each with tiny changes. In several documentaries I've worked on, we were absolutely confident of "picture lock" but still ended up make 10 or 15 more versions, each requiring a time-consuming render as a deadline loomed.

This situation is common in film post production, even before the digital age. To show how you never really have "picture lock", Empire Strikes Back was already released in 70mm (in 100 theaters) and George Lucas decided he wanted some editing changes before the 35mm print was released. So they did additional shots, re-edited it, pulled the 70mm prints, added those edits, and re-released it. At least with Vimeo we can upload a new edit and the URL doesn't even change.

OTOH if the export codec is ProRes, in my tests pre-rendering didn't make much difference. It just wasn't worth it in that case. So the potential benefit (if any) of pre-rendering varies based on export codec, timeline complexity, and whether multiple subsequent exports will be done after the pre-render.
The administrator has disabled public write access.
  • Page:
  • 1
  • 2