Welcome, Guest
Username: Password: Remember me

TOPIC: Solutions for laggy playback when on a deadline

Solutions for laggy playback when on a deadline 07 Oct 2019 06:18 #101857

  • Suzero
  • Suzero's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 104
  • Thank you received: 4
  • Karma: 0
I have taken over an edit from a director. It's a 90 minute documentary with mostly 4K XAVC-i interviews and all sorts of SD/HD archival footage in various formats, being edited in a 2K timeline. We are working from a Pegasus RAID via thunderbolt.

It is, I suspect, the XAVC-i footage causing my iMac (see specs in attached image) to lag. As I am building layers of effects / title animations etc. it is driving me nuts and slowing down my work process to the point where I will not meet the deadline.

So I bit the bullet and started creating optimised media overnight (hoping that I have enough storage). Proxies is not an option as I am too near delivery time and need high quality footage to work with now for effects etc.

I selected the most recent project and told FCP to transcode to optimised. 12 hours on and it is still working. So I have a couple of questions:

1. If I stop the transcoding to continue editing today, can I start the transcode again during lunch break and will it pick up where it left off?

2. Is there a way to see how long the entire transcoding will take? The Background Tasks window only shows progress per clip that is being transcoded - not how many more there are left to go.

3. What kind of Mac can playback XAVC-i 4K smoothly without needing to transcode? I know my MacPro (2013) can't. Will an iMac Pro do the trick? I may have to start saving up.

Thanks for any advice!


Screenshot2019-10-07at08.04.51.png
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 07 Oct 2019 09:10 #101860

  • Suzero
  • Suzero's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 104
  • Thank you received: 4
  • Karma: 0
Another question:

I can see that certain shots have been optimized in the Optimized Media folder, but if I select the clip in the project > Reveal in Browser > Reveal in Finder - it still points to the MXF instead of the MOV.

How can I get FCPX to look at the MOVs? After all, that's the whole point of transcoding, right?
Last Edit: 07 Oct 2019 09:11 by Suzero.
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 07 Oct 2019 10:32 #101861

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1247
  • Thank you received: 258
  • Karma: 22
Suzero wrote:
...90 minute documentary with mostly 4K XAVC-i interviews and all sorts of SD/HD archival footage in various formats, being edited in a 2K timeline. We are working from a Pegasus RAID via thunderbolt....It is, I suspect, the XAVC-i footage causing my iMac (see specs in attached image) to lag. As I am building layers of effects / title animations etc. it is driving me nuts and slowing down my work process....So I bit the bullet and started creating optimised media overnight (hoping that I have enough storage). Proxies is not an option as I am too near delivery time and need high quality footage to work with now for effects etc.

Some intraframe formats decode quickly in an NLE. Sony's XAVC-I does not, which is shown in their white paper XAVC Specification Overview on figure 9 (see attached): www.xavc-info.org/xavc/share/data/XAVC_S...nOverview_Rev2_2.pdf

I have more experience with 4k XAVC-S which is very slow to decode, even on a 10-core Vega64 iMac Pro.

Suzero wrote:
..I selected the most recent project and told FCP to transcode to optimised. 12 hours on and it is still working.

1. If I stop the transcoding to continue editing today, can I start the transcode again during lunch break and will it pick up where it left off?

Yes. If you just re-select all the clips it will not re-transcode those already done, only the remaining clips needing transcoding.

Suzero wrote:
...2. Is there a way to see how long the entire transcoding will take? The Background Tasks window only shows progress per clip that is being transcoded - not how many more there are left to go.

If you twirl down the disclosure triangle on the background tasks window it will show you a scrollable list of the remaining files. It doesn't show how long each clip is but by the number of clips remaining you can roughly approximate time to go.

Suzero wrote:
...3. What kind of Mac can playback XAVC-i 4K smoothly without needing to transcode? I know my MacPro (2013) can't. Will an iMac Pro do the trick? I may have to start saving up....

I haven't tested 4k XAVC-I extensively but Sony's white paper implies it is slow to decode. If you could upload or point me to some 4k XAVC-I test footage encoded like yours I can test it on several different Macs.

However in general the most reliable method of avoiding cases like this is acquiring in ProRes. My documentary team now uses Atomos recorders on all our cameras. It chews up lots of space but transcoding to optimized media takes both space *and* time. With ProRes acquisition, the moment you offload you can start editing. Usually no proxies are required, provided your disk subsystem is fast enough.

I don't know if Quick Sync accelerates the format of XAVC-I you are using. Some of those are 4k 4:2:2 10-bit. Your 2014 iMac uses a "Haswell" generation CPU. Quick Sync was significantly improved starting with Skylake which was used in the 2017 iMac 27. I have one of those so I could test the 4k XAVC-I material if I had some. In general the 2017 iMac is about 2x faster on XAVC-S than the 2014 and 2015. However IMO it's still not fast enough for large 4k XAVC-S projects without generating proxies or optimized media.
Attachments:
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 07 Oct 2019 11:46 #101863

  • Suzero
  • Suzero's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 104
  • Thank you received: 4
  • Karma: 0
Thank you so much for taking the time to respond in such detail! It's a relief only to know it's not just me getting increasingly impatient :-)
I am rushing now with the edit, but when I get a chance I'll send you some footage. Thanks for the offer!
Incidentally, footage plays back OK in the beginning, but as editing progresses everything slows. Even now having transcoded most of the shots! I regularly quit and restart FCP because of this but not much joy.
Audio also drops out during playback sometimes.
I am still not sure that FCP is looking at the optimized files...
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 07 Oct 2019 12:24 #101866

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1247
  • Thank you received: 258
  • Karma: 22
Suzero wrote:
... footage plays back OK in the beginning, but as editing progresses everything slows. Even now having transcoded most of the shots! I regularly quit and restart FCP because of this but not much joy.
Audio also drops out during playback sometimes. I am still not sure that FCP is looking at the optimized files...

You can try these things. Of course in the middle of a deadline you must be careful:

- Make sure you have plenty of free space on all drives
- In general run the latest version of macOS and FCPX but be cautious about updating in the middle of a project.
- Run the latest version of Apple Pro Video Formats. Starting with Mojave this is in System Preferences>Software Updates.
- Turn off FCPX background rendering. Delete all render files. Do a one-time manual timeline render by selecting all clips with CMD+A, then CTRL+R. That will gradually become unrendered as you work but late in an edit this can save time and avoid problems with background rendering.
- Make sure your library and cache are on a different drive than your media. Every edit is stored in SQL tables inside the library and FCPX is constantly scanning those and inserting new rows. The I/O profile is small and random and conflicts with the large sequential reads needed for media.

By default transcoded media inc'l proxy and optimized is stored inside the library which is not optimal from a performance standpoint. You can move this by defining a new storage location in the Library Inspector under Storage Locations>Modify Settings. Giving a new media location then doing "consolidate" will move all proxy and optimized media to that location. This in turn makes the library small, allowing it to be placed on a different drive from your media drive. It does not move any media imported with "leave files in place".

Re is it really using the optimized files, normally that is very reliable. Unfortunately there is no quick approved method to test this. However you could theoretically pick one clip and move it from the library to another hard drive. FCPX would revert to the original media. You'd then skim the clip in the browser to observe playback performance. It should be much slower. You could then copy the clip back to the original library or designated storage location, whereby skimmer performance on that clip should be much faster. Do this at your own discretion.

A greater likelihood for the progressive slowdown might be effects. Certain effects like stabilization, skin smoothing, video noise reduction, etc. are very CPU intensive. Ideally those should be delayed until very late in the edit.

Although optimized media is very fast from a CPU performance standpoint, the files are large and require good I/O performance. Relative to common H264 codecs they are about 5x or 6x the size. I don't know relative to XAVC-I.
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 07 Oct 2019 15:01 #101867

  • arc
  • arc's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 57
  • Thank you received: 9
  • Karma: 1
joema wrote:
Suzero wrote:
...90 minute documentary with mostly 4K XAVC-i interviews and all sorts of SD/HD archival footage in various formats, being edited in a 2K timeline. We are working from a Pegasus RAID via thunderbolt....It is, I suspect, the XAVC-i footage causing my iMac (see specs in attached image) to lag. As I am building layers of effects / title animations etc. it is driving me nuts and slowing down my work process....So I bit the bullet and started creating optimised media overnight (hoping that I have enough storage). Proxies is not an option as I am too near delivery time and need high quality footage to work with now for effects etc.


I don't know if Quick Sync accelerates the format of XAVC-I you are using. Some of those are 4k 4:2:2 10-bit. Your 2014 iMac uses a "Haswell" generation CPU. Quick Sync was significantly improved starting with Skylake which was used in the 2017 iMac 27. I have one of those so I could test the 4k XAVC-I material if I had some. In general the 2017 iMac is about 2x faster on XAVC-S than the 2014 and 2015. However IMO it's still not fast enough for large 4k XAVC-S projects without generating proxies or optimized media.

Good idea. I would not mind doing some playback testing of the XAVC video files if someone could upload five or six 20 second long clips.
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 08 Oct 2019 06:48 #101893

  • Threefold Films
  • Threefold Films's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 37
  • Thank you received: 3
  • Karma: 1
Just wanted to chime in from someone who deals with XAVC-I every day in 4K. this does not sound like a typical experience. XAVC-I performs similarly to prores at 4K in our experience. Your machine should find this no trouble at all especially if the viewer is set to "better performance" I can play XAVC-I on a 2011 MBP so not too sure what's going on. XAVC-L however is a completely different kettle of fish, and performs similarly to H264 at 4K. Do you have another machine you can move the raid to, just to rule out something funky there?
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 08 Oct 2019 06:52 #101894

  • Threefold Films
  • Threefold Films's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 37
  • Thank you received: 3
  • Karma: 1
"it still points to the MXF instead of the MOV" that's interesting, I'll double check but I thought if you import XAVC into FCP correctly it created .movs of the media rather than keeping the .mxf
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 08 Oct 2019 10:42 #101896

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1247
  • Thank you received: 258
  • Karma: 22
Threefold Films wrote:
Just wanted to chime in from someone who deals with XAVC-I every day in 4K. this does not sound like a typical experience. XAVC-I performs similarly to prores at 4K in our experience....

Thanks for that. My experience with 4k All-Intraframe material from Canon is also very fast. We would need an unaltered short sample clip from the OP's camera to verify encoding and characteristics. Maybe it's really XAVC-L, or there are unknown system bottlenecks like a USB 3 drive using a hub or non-spec cable making it work at USB 2 rates.

If the OP could run Blackmagic speed test on each drive in the system inc'l boot drive and also state % free space on each drive that would also help.
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 08 Oct 2019 11:05 #101898

  • Threefold Films
  • Threefold Films's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 37
  • Thank you received: 3
  • Karma: 1
Agree... I don't think it's the XAVC-i linking to .mxf rather than.mov isn't how FCPX imports that footage, so it sounds like ... would be interested to know what the other codecs sitting on the timeline are too....
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 09 Oct 2019 03:44 #101928

  • Suzero
  • Suzero's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 104
  • Thank you received: 4
  • Karma: 0
Thanks for all the replies so far!

I am still in the throes of finishing off this project, so I can't share any footage for testing ( and would need to ask permission first).

To clarify: playback has been OK sometimes, but at other times things slow down to such an extent that dragging an inspector value lags enormously. That's the most frustrating thing as it is really slowing me down.
What I also find weird is that even in parts of the timeline that have been rendered, on playback, the audio will just drop out here and there whilst the video plays back smoothly.

However, I think I may have found a / one of the culprit(s) for the lag: HUGE TIFF files that the director had imported and edited into the timeline. Some were 300dpi and over 10000 pixels wide. I remember this used to cause problems in FCP7 too! I am currently working on getting those out of the timeline, and even out of the Library, in the hope that that will help speed things up.
The administrator has disabled public write access.

Solutions for laggy playback when on a deadline 09 Oct 2019 10:24 #101939

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2780
  • Thank you received: 367
  • Karma: 34
Convert to 72dpi png, you should be fine. Those are ridiculously large files.
The administrator has disabled public write access.