Welcome, Guest
Username: Password: Remember me

TOPIC: Library Incredibly Slow to Load

Library Incredibly Slow to Load 10 Sep 2019 12:52 #101362

  • richcamp
  • richcamp's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 21
  • Karma: 0
I have been cutting a 12 minute short film, there's a decent amount of clips - two cameras, 8 tracks audio, some sfx and music but honestly nothing crazy. The library is 22gb - I deleted the proxy footage and the original media is located on an external drive. Opening the library even without any media connected is so painfully slow - it takes a few minutes before it even can load. FCPX is unresponsive and in a crashing state the whole time. Then once it's open, every click is painfully slow. Loading a project takes a few minutes, etc.

I've attempted resetting preferences, clearing created media, but nothing seems to help. Any insights on why this library is so slow and how I can get it to run a bit smoother?

I work in post professionally but at work I use Avid, I've been using FCPX for side projects for years and know I've seen this before FCPX where the finishing of a project slows a library substantially but it's honestly pretty unusable. I know at work Avid runs smoothest without media attached so I'd expect FCPX without media would be quicker. I am working off a 13" MBP but since there's no media I'm not actually doing anything and just opening a library, I'd still expect it to open and be manageable at least.

Any help is appreciated! Thank you!
The administrator has disabled public write access.

Library Incredibly Slow to Load 10 Sep 2019 13:46 #101365

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1190
  • Thank you received: 245
  • Karma: 21
I've edited a documentary containing 8,500 4k H264 clips, 220 camera hours and 130 multi-camera interviews in a 20TB library on a 2017 iMac 27 and haven't seen this. In this case media was on dual redundant OWC 32 TB OWC Thunderbolt arrays in RAID-0.

There is one known case if you have AVCHD material that was copied out of the bundle and imported using "leave files in place", it can create extreme amounts of unnecessary IO and many library functions can be slow. Do you have any MTS or similar tree-oriented media that was imported that way?

FCPX may use Spotlight indexing to locate media and if these are out of date it might slow down certain phases. Try running Disk Utility First Aid on all volumes, then rebuild Spotlight indexes on all volumes: support.apple.com/en-us/HT201716

All volumes should be on Mac HFS+ or APFS volumes not on exFAT or NTFS. If any of your material is on a NAS state the specifics.

Make sure you are running the latest version of FCPX and macOS and also the latest version of Apple Pro Video Formats. Starting with Mojave this is updated in System Preferences>Software Update.

Several of the newer USB-C SSD devices are extremely sensitive to cable type. E.g, Samsung T5, Sandisk Extreme Pro, etc. If a non-OEM cable is ever used the device can fall back to USB 2.0 rates, which is 20x slower.

As a sanity check run Blackmagic speed test on each disk volume and verify the I/O performance is within the expected range.

I normally don't open libraries with external media disconnected, so I'm not sure the expected performance implications. When I do that by accident I don't remember super-long library loading times.

Make sure all disk volumes have plenty of space. I suggest your library be configured with cache in a separate location. In FCPX click the library, then in Inspector>Modify Settings, set cache to a folder on a separate drive. This splits I/O load and enables easy deletion of the cache from Finder if you ever suspect stale or corrupt cache files.

Each project or project snapshot you save creates a separate CurrentVersion.fcpevent file within the library. Within that file are several SQL database tables. If you have a large number of snapshots or projects that can slow FCPX library loading time, but it has "deferred loading" logic which tries to minimize this. In general I've not seen it really bad even with many dozen project snapshots.
The administrator has disabled public write access.

Library Incredibly Slow to Load 10 Sep 2019 15:31 #101367

  • VidGreg
  • VidGreg's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 929
  • Thank you received: 185
  • Karma: 28
Hi RichCamp
What is the model of MBP 13" do you have and how much RAM is installed?
In addition to what Joema has suggested…
I would open Activity Monitor and click on the "Memory" tab and at bottom check on the amounts of RAM being used. Pay attention to the "Swap Used" numbers and if high, then you are potentially seeing one possible reason.
Do you use Chrome? This has been known to cause high RAM usage and computers to really slow down while running Chrome.
Quit any apps you do not need to be running.
The other tabs in Activity Monitor can indicate what is using CPU, Memory, Disk activity. This may help pinpoint what resources are being used.
What format and resolution of media can also be an issue. Have you tried Proxies?
How are your external drives connected? USB/ TB? If USB, is it USB 3.1 gen2 ? Does your drive support UASP or older bulk transport?
Joema's comment on cables is a good one as I have experienced cable issues even with device supplied cables. Also don't forget that different cables have different speed ratings and if you use a USB 2 cable, it will not run at USB 3 speeds.
Have you tried to fully shut down and then restart the Mac?

Hope this Helps, Greg
The administrator has disabled public write access.

Library Incredibly Slow to Load 10 Sep 2019 15:38 #101368

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1190
  • Thank you received: 245
  • Karma: 21
VidGreg wrote:
....
Do you use Chrome? This has been known to cause high RAM usage and computers to really slow down while running Chrome...Quit any apps you do not need to be running...

I forgot to mention this, thanks VidGreg. It appears Chrome under some conditions saturates the macOS VideoToolBox framework, thereby locking out FCPX. This in turn causes extremely slow FCPX performance. It is not limited to FCPX - any NLEs or transcoders that rely on VideoToolBox for encoding, decoding, and transcoding are potentially affected.
The administrator has disabled public write access.

Library Incredibly Slow to Load 10 Sep 2019 15:43 #101369

  • richcamp
  • richcamp's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 21
  • Karma: 0
Thank you! I am going to test these things out tonight.

I do not use Chrome, I have tested with everything closed besides FCPX. I used proxies for the edit then cleared them when I finished the edit. What's confusing me is this Library is just incredibly slow without any media attached, without attempting to play media, its just opening and loading a projects etc. I know that a MBP 13" (2017 16gb ram dual i7) isn't optimal but it's also not like I'm trying to do anything beyond opening the Library which has me questioning if there could be a corruption or cache issue.

I'll update after I test out these things tonight! Thank again!
The administrator has disabled public write access.

Library Incredibly Slow to Load 10 Sep 2019 22:00 #101374

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1190
  • Thank you received: 245
  • Karma: 21
richcamp wrote:
...What's confusing me is this Library is just incredibly slow without any media attached, without attempting to play media, its just opening and loading a projects etc...

By "without any media attached", do you mean it's only showing red "missing media" clips? I assume it was slow before and this is only a test. However loading a library with missing media isn't a good test because FCPX is doing some extra work to try and validate the links. There is a two-step validation procedure: It uses the pathnames stored in the library and only if that fails does it tries an inode lookup. I don't think it probably makes much difference but testing with disconnected media in an already anomalous situation might be misleading.

Sometimes there can be a big difference in how the Event Browser populates in filmstrip mode vs list mode. If there is something wrong with the cache or if you have "in place" AVCHD .MTS files, thumbnails can populate very slowly.

Besides the previous recommendations, I suggest deleting all render files via Files>Delete Generated Library Files. Unfortunately there is no exposed way to delete the cache but if the library storage properties have cache set to a specific folder you can delete the bundle object in that folder. If you are not certain about AVCHD media you could do a Finder search for M2TS, MTS and M2T files. If those exist exposed outside the AVCHD package they might have been imported using "leave files in place" which can create a huge performance penalty.
The administrator has disabled public write access.

Library Incredibly Slow to Load 10 Sep 2019 22:18 #101375

  • richcamp
  • richcamp's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 21
  • Karma: 0
Thanks! I am using all Blackmagic Pocket 4k Prores media, no AVCHD or MTS.

I deleted all generated media, which helped a bit. Then I took your suggestion to move cache storage outside the Library and that was a HUGE help. It was an immediate speed up after that was done, which is great. Its running a bit smoother, I am going to try a few of the other suggestions and look further into it. I do have a LOT of project snapshots, as I cut each scene on its own and did a snapshot for each version (RC1, RC2, FC, etc). Any suggestions on maintaining those in the future?

Is there any benefit to having multiple Libraries for different types of content? Like a media library, a project library, so I can have the least amount of contents open at once? I also know that sometimes the Libraries can get messy with a lot of aliases linking to external footage or internal media, etc. I use to do Copy to Library but that got really confusing so I've moved to Leave in Place so I can sort it as I want.
The administrator has disabled public write access.

Library Incredibly Slow to Load 10 Sep 2019 23:14 #101376

  • joema
  • joema's Avatar
  • NOW ONLINE
  • Platinum Boarder
  • Posts: 1190
  • Thank you received: 245
  • Karma: 21
richcamp wrote:
... I do have a LOT of project snapshots, as I cut each scene on its own and did a snapshot for each version (RC1, RC2, FC, etc). Any suggestions on maintaining those in the future?

Is "a lot" 20 or 100? Each snapshot is a separate CurrentVersion.fcpevent file, each one containing several SQL tables with all your edits, metadata and pointers to all media. Normally having 20 or so is no problem but if the edit is complex (say many hundreds or thousands of actions), then each snapshot is another copy of that. The effect is multiplicative - it's project complexity times number of snapshots.

FCPX has some kind of "deferred project loading" logic whereby it will sometimes not enumerate and display all snapshots. You see this when doing a query or click on a menu then it starts enumerating the snapshots and displaying the thumbnails. It is probably trying to minimize overhead.

I have used very large libraries and vastly complex timelines, probably containing thousands of edits. In these FCPX seems more likely to slow down library loading when creating over (say) 30 snapshots. By slow I mean 20 seconds vs 5 seconds, but I'm on an iMac Pro with a Thunderbolt array. Once loaded performance seems mostly OK.

If you can delete any snapshots you don't need, that would help. If you have a large # you need, I don't think moving them to another event within the same library would help, but you could try it. When clicking on the top-level library it will try to enumerate them no matter what event they are in since the search scope is library-wide.

You could create an archive library and move the less-needed snapshots there (but note below caution).. For external media FCPX will create symlinks, not move the actual data (provided you don't consolidate it). If on the same volume it will in addition use hard link optimization where needed to further minimize space consumption. As always make sure you have good backups before doing anything like this.

If you are using "lean libraries" it is very easy to make multiple file-level backups of the library. In Finder, just right-click and select "duplicate". A "lean library" has all media and cache external and will be small, even if linked to 10,000 clips and many projects with thousands of edits.

richcamp wrote:
....Is there any benefit to having multiple Libraries for different types of content? Like a media library, a project library, so I can have the least amount of contents open at once? I also know that sometimes the Libraries can get messy with a lot of aliases linking to external footage or internal media, etc. I use to do Copy to Library but that got really confusing so I've moved to Leave in Place so I can sort it as I want.

In general I think people are too quick to create separate libraries, which in turn puts up "data fences" between commonly-needed assets. If you are under 10,000 clips and 30 terabytes I wouldn't normally consider splitting a library just for performance reasons. ProRes is less dense, hence less clips so maybe 40-50 terabytes in that case.

Yes I previously suggested maybe creating a library for extra snapshots but that is a special case. Normally people don't have or need 100+ snapshots. If your system is super slow with 20 or 30 snapshots, examine your storage subsystem. It normally shouldn't be that slow. For projects of this scope you should normally be on a RAID array of some type.

If you are using a NAS, remember unless you are on 10 gigabit ethernet that can be quite slow (maybe less than 100 MB/sec).

It's important to visualize the various concurrent I/O streams employed by FCPX. Besides one stream of large sequential reads for media, there are two other streams of small random I/Os - one for cache and the other for the library database. Ideally have cache+library on a fast SSD drive and media can be on a spinning drive or RAID array. Spinning RAIDs are not good at small random I/Os, thus the separate SSD for library/cache. If using a lean library both it and cache can be on the same SSD. If using only spinning single drives consider putting them on separate drives with media on a RAID.

Never copy projects and clips between libraries unless they are in a "transfer event". Sam Mestman discusses this from 06:30 to 11:00 in the below video, but you may want to watch Sam's whole talk from 02:00 to 11:00. He demonstrates this on a Lumaforge NAS but it's not unique to a NAS.

hazu.io/pixelcorps/fcvug-7
The administrator has disabled public write access.