Folks, I am trying to figure out why there's such a huge disparity between copying a library from one drive to another using the Finder vs. using FCP X to copy content.
Two USB drives connected to the same computer. I copy one library (1TB) from one drive to the other using the Finder and it takes somewhere between 2.5-3 hours.
Copying an event from the same starting USB drive to the same destination USB drive, but with the same content and using FCP X to create and copy the data takes well over 10 hours.
This has been happening no matter what kind of tests I do. I understand that FCPX has to build it's database draw waveforms, etc. and therefore would be a little slower, but these time differences are huge.
Any ideas folks? Managing large amounts of data using FCP X and distributing to other editors is a significant workflow slowdown even when letting things run overnight; being that copying is still not done often times.
The administrator has disabled public write access.
Copying takes significantly more time inside FCP X vs. Finder - why?
30 May 2019 21:17 #100062
....trying to figure out why there's such a huge disparity between copying a library from one drive to another using the Finder vs. using FCP X to copy content.... I copy one library (1TB) from one drive to the other using the Finder and it takes somewhere between 2.5-3 hours.
Copying an event from the same starting USB drive to the same destination USB drive, but with the same content and using FCP X to create and copy the data takes well over 10 hours...
I just tested this on macOS 10.14.5 and FCPX 10.4.6 on an iMac Pro, copying between the internal SSD and a four-drive RAID-0 Thunderbay 4, and I get the same performance whether using Finder or FCPX.
Ingesting from RAID array to within a library on the iMP SSD, I get about 850 MB/sec on both input and output. Copying via Finder I get the same performance.
I then created a managed library on the RAID array and drag/dropped the event from the 1st library on the SSD to the RAID system, and FCPX did about 850 MB/sec, so no difference in speed between Finder and FCPX in this simple test.
If it's taking you 3 hr or 10 hr to move 1TB that is an average speed of 93 MB/sec for 3 hr and 28 MB/sec for 10 hr.
93 MB/sec is about the expected performance of a 5400 rpm bus-powered USB 3.0 spinning drive. However 28 MB/sec is much slower than expected. Are these drives formatted HFS+ or exFAT? I haven't seen much performance difference between those, but I haven't tested FCPX extensively on exFAT.
Another possibility: my test case was a simple test library with a few large files. A well-used FCPX library can contain thousands of small files: plist, waveform, thumbnails, render files, etc. When copying a well-used event you are possibly copying thousands of render file fragments, media (or symlinks), and possibly many waveform, plist and render files. It's for this reason that duplicating a library via Finder right click>duplicate can take a while, even though the total # of megabytes is modest. However from this aspect alone, I'd expect a copy via FCPX to be no slower than Finder.
Yet another possibility: the SQL database is in a single file for each event called CurrentVersion.fcpevent, and it may contain many thousand rows. Those rows contain all your edits, metadata about the effects stack, and pathname and inode info about each media file. When copying the containing library at the Finder level, CurrentVersion.fcpevent is a single file. But when FCPX copies that it must insert each row via a separate operation, because those rows contain specific information about the new file pathnames on the destination drive. The tables have indexes so each insert operation also entails index overhead. Just a guess but that is a possible factor.
It might be possible to move your media within FCPX to a "transfer" database on the same drive. That will use hard links and should be fast. Then close that library and copy it via Finder to the destination drive. That bypasses any internal FCPX SQL operations. Sorry I don't have time to test this right now.
Edit/add: it's possible to use Activity Monitor to approximate the IO size during the "fast" vs "slow" tasks and maybe get a hint about why it's slow. E.g, in the Disk tab, if you divide Data read/sec by Reads in/sec, that will roughly indicate the IO size for reads. Likewise dividing Data written/sec by Writes out/sec will roughly show the IO size for writes.
If the average IO size during the slow FCPX media management is much smaller than those during the Finder copy, that may indicate FCPX is doing lots of small IOs for database updates.
Thanks for the reply and your details explanation has things that I have considered as well. I suspect it's the SQL database. I just find this really tough to deal with. I've always noticed that copying using FCP X is significantly slower than doing it in the finder and I haven't found a solution. The drives I was using are 5400 rpm bus powered drives. I should do more tests with my RAID. However, I've been noticing this for several years with several versions of FCP X, macOS, and various drive setups. I wish the coping was at least close, but it never is.
Will have to do some tests and document. It's like the difference between a Mac with and a Mac without QuickSync but with multicore Xeon chips. They're not even remotely close in compressing H.264 even though the latter has Xeon chips and more cores.
The administrator has disabled public write access.