I filed this bug report today. Due to the severity of the problem I'm classifying it as a bug not an enhancement request. This isn't some new issue -- it's been like this a long time, I just now got around to filing this.
Title: Poor relink file search efficiency and UI behavior
Description: There can be a severe performance and UI problem when relinking files on a large volume. This is composed of four elements (1) Very slow "looking for matching filenames" phase, (2) UI defect which does not allow the user to explicitly start the file search (3) UI threading defect which does not properly reflect status but locks up window with a beach ball, (4) UI defect which does not properly handle user cancel request.
Result: UI locks up for long periods -- up to 15-20 minutes, even on a Thunderbolt 2 RAID-0 array. This condition self-initiates while the user is simply navigating to the correct folder in the "Relink Files" Finder window, before the "Choose" button is selected. Thus the user's attempts to provide a narrower search scope are often futile.
As discussed in the Apple article "Relink clips to media files", it is possible and supported to relink files having different resolutions and codecs and clip length than the original:
However this usually requires navigating to the file in question. With the above behavior this becomes almost impossible on a large volume because the relink search often begins and locks up the UI before the user clicks the "choose" button. This can incur a 15 min wait per file to relink each one.
This behavior has long existed and been observed on various FCPX platforms and hard drives. However I proactively ran Disk Utility First Aid, also rebuilt all Spotlight indexes to rule out problems from disk structures or stale indexes.
This is interesting, but you're probably going to have to do some more tests to isolate whether it is indeed a hardware problem or not. Even if first aid doesn't turn up anything, the complexity of different RAID systems and controllers means there are a lot of other variables. For example, you'll have to get a hold of some other large RAIDS that connect through thunderbolt in the same way and do the same things to see if you're getting the same performance issues.
I say this because I've personally not seen these multi-minute lock-ups that you've described, and we routinely do relinking on very large volumes over both thunderbolt, 1 GbE, and 10 GbE. I've gotten some beach balls here and there, but they don't last that long. It's possible that there are bugs that exist with certain configurations, and this needs to be isolated.
haysoos123 wrote: This is interesting, but you're probably going to have to do some more tests to isolate whether it is indeed a hardware problem or not....you'll have to get a hold of some other large RAIDS that connect through thunderbolt in the same way and do the same things to see if you're getting the same performance issues....
I have an office full of various Thunderbolt 2 RAID arrays, including an 8TB RAID-0 SSD array, and multiple 16 and 32TB RAID-0 arrays, plus side-by-side 2015 and 2017 iMac 27s. I've seen this on several different unique hardware, I/O systems and video data sets, including a 12-core D700 Mac Pro. I tend to doubt it is hardware related.
haysoos123 wrote: ...I've personally not seen these multi-minute lock-ups that you've described, and we routinely do relinking on very large volumes over both thunderbolt, 1 GbE, and 10 GbE. I've gotten some beach balls here and there, but they don't last that long. It's possible that there are bugs that exist with certain configurations, and this needs to be isolated.
The problem seems to grow exponentially worse as library size increases. I'm not sure what the threshold is, but maybe you are below it, or maybe it is codec specific, or maybe it only happens if using non-managed libraries and external proxies. The library I'm now using is all H264 4k, 5218 clips, 187 hr of material, the media is 9.49TB and the proxies 3.81TB. However I saw it at significantly smaller library sizes.
With that size of library, I'm not surprised at all. It might even be related to the long-standing issue of sluggishness once a project gets large. Pretty sure you've seen that with your library sizes... not sure what it could be, but I would bet it's related to the way it uses memory. So from my experience, I don't think it has much to do with the size of the drives but rather the size of the libraries. We surely did have these same problems in FCP7 days, when a project would get to a certain size and become very unstable.
Not sure if you've been able to do this before, but maybe Apple will contact you directly about the issue if you can demonstrate it. I was able to do a screenshare with the Apple techs and show them the sluggishness issue (which has improved since then but isn't yet ideal). I think the sheer number of clips you have in one library is related to that. I'd also try to make a dummy project with about the same number of clips/timelines, but use other types of codecs... I think H264 4k is very taxing computationally (but I'm not sure it makes too much of a difference with memory usage). I think they only thing you can do for now is look at a workflow where you're not constantly having to relink clips.
PS, what I've still observed is that timelines with lots of titles or use of adjustment layers tend to be a lot worse. So libraries with lots of these sorts of timelines: long, with many titles, will bog the whole thing down. Unfortunately that doesn't really help if you have to use them, though.
haysoos123 wrote: With that size of library, I'm not surprised at all. It might even be related to the long-standing issue of sluggishness once a project gets large. Pretty sure you've seen that with your library sizes... I don't think it has much to do with the size of the drives but rather the size of the libraries....
As I previously stated, I see this on smaller libraries. I just tested a cut-down version of the above library with less than 1/2 the content, plus no proxies: 2,487 clips, 85 hr, 4.42TB. In this era, that's not gigantic, esp. on high-end hardware.
On that smaller library, when relinking a single file it still has the same problematic behaviors: UI does not wait for user folder navigation and pressing "Choose" button to launch the relink search. Once the spontaneously-initiated search begins, the UI locks up with a beach ball for several minutes. Thus it is not possible to narrow and expedite the search scope by navigating to a smaller folder tree before pressing "Choose" and starting the search for files to relink.
On this smaller library, the FCPX UI stays locked up for about 5 min vs about 15 min on the larger data set. Neither is appropriate or acceptable -- it is a clear software defect. Apple's test cases for this functionality may have only included tiny "toy" libraries. I cannot imagine a software test engineer who would knowingly allow this behavior to pass.
Re memory, it's not a memory leak. The problem appears to contain two main elements: (1) UI is not handling user events correctly and autonomously launching the relink search (2) Algorithm when searching for compatible files is highly inefficient. It is doing some kind of brute-force lookup that scales poorly to larger library sizes.
Apparently upon clicking Relink, the UI allows folder navigation for a brief period. You can spin down disclosure triangles, but if you either (a) double click on a folder to open it, or (b) wait more than a few seconds it will spontaneously enter the beach ball mode for a long time. While the behavior manifests consistently, the severity is erratic. Even on the above smaller test case if I don't immediately click the file after the 5 min beachball, it will re-enter the beachball state for much longer.
If you are relinking a large number of files, it's less an issue since you pay the 5-15 minute wait penalty once, then they all (hopefully) relink. However if you are relinking several individual files in different passes, it is impractical to use.
You won't see enhancement requests for issues like this in any FCPX "Top 100 Requests". Those are typically superficial UI-centric items like color-coded video lanes and a scrolling timeline. I'd like those too, but in the meantime I can swipe my mouse to scroll the timeline. I cannot effectively relink files in this case. It is a very serious problem which impacts large professional projects.