fcp.co logo transparent
fcp clapperboard
Welcome, Guest
Username: Password: Remember me
14 Nov 2020
Moving over to a new system, we couldn't take the old avatars - so please upload a new one!
  • Page:
  • 1

TOPIC: Waveforms very slow to rebuild

Waveforms very slow to rebuild 30 Oct 2020 14:28 #110654

I am running FCPX 10.4.6 on a MacMini 2019 16gb running Mac OS 10.14.6 (Mojave)
Currently I have not made the suggested updates that Apple has for me: ProRes codecs, Final Cut Supplemental update, etc, because I just reinstalled clean the OS and FCPX. I had upgraded to 10.4.9 and a 3-year project started having issues with missing audio on some files from certain cameras, plus bugs such as clips that had the Color Wheels applied claiming to be "Missing Plugin" and then upon restart FCPX being back to normal. So I decided 10.4.9 is buggy and especially dangerous if I am mid-project. Since I had done almost nothing after upping to 10.4.9, I did a clean reinstall of the OS and FCP 10.4.6 and also deleted all audio Peak data.
So now I am taking notes on every thing I notice, using Time Machine and only upgrading/installing updates if I feel its safe and then watching to see if something happens. Attached is an image of the dark red Audio track.
But for now, my issue is that the computer is VERY slow to rebuilt waveforms. Clips have audio alright, I hear it upon playback or skimming, but in the browser the audio track appear as dark red, with no Peak data. I took note of which clips have this issue, and they are coming from two or three different cameras, and not all files form those cameras have this issue: some have waveforms all OK. And as I wait, some ones that were missing Peak data are getting that too, so I know it is rebuilding Peak data, but VERY slow. It is a large project, with thousands of clips, so this must be the issue, right? Any ideas about this? Much appreciated, I am new to fcp.co but have read several forum chats before and its always been helpful.
Thank you!
Attachments:

Please Log in or Create an account to join the conversation.

Waveforms very slow to rebuild 30 Oct 2020 15:01 #110655

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1775
  • Karma: 27
  • Thank you received: 395

Alan Langdon wrote: .. I decided 10.4.9 is buggy and especially dangerous if I am mid-project. Since I had done almost nothing after upping to 10.4.9, I did a clean reinstall of the OS and FCP 10.4.6 and also deleted all audio Peak data....Attached is an image of the dark red Audio track...But for now, my issue is that the computer is VERY slow to rebuilt waveforms. Clips have audio alright, I hear it upon playback or skimming, but in the browser the audio track appear as dark red, with no Peak data....


The current version is 10.4.10. It has significant performance improvements over 10.4.6. In general 10.4.10 is not buggy but if you have old plugins which have not been updated, those can cause problems.

While it's not usually a good idea to update in mid project, this can often be safely done with the proper procedures. You simply need to have backup copies of your libraries and in Finder duplicate and rename the 10.4.6 version of Final Cut Pro.app in /Applications. That allows an easy safe roll back if any problems.

All plugins should be first updated, *especially* those from CoreMelt. Here are their installers: coremelt.com/pages/downloads

Catalina is not required to FCPX 10.4.10, Mojave is OK.

If your libraries are very large due to containing media or cache, then duplicating them is difficult. The solution is only use "lean libraries" which have external media and external cache. Cache can be placed in a user-designated folder by using the FCPX library inspector, Storage Locations>Modify Settings>Cache.

Having cache external accomplishes several things:

(1) Keeps your library smaller, thus allowing easier duplication for backup.
(2) The small library can be more easily located on an SSD which is much faster for the random I/O required for the library database.
(3) Puts all cache items external for possible placement on an SSD or other fast drive. Since waveforms and thumbnails are stored in cache, this can really help performance.
(4) External cache allows easy deleting of the cache bundle from Finder. This sometimes fixes problems with black thumbnails or red waveforms. The waveforms and thumbnails are not deleted by the command File>Delete Generated Library Files>Delete Render Files>All. However if cache is external you can safely shut down FCPX and delete the cache which is in a folder you specify. You delete the bundle named LibraryName.fcpcache. It will be automatically rebuilt.

From a performance standpoint you don't want the library on the same drive as media. Library and cache can be on the same drive if it's a fast SSD.

Disable background rendering in FCPX preferences. If needed you can render the timeline by selecting all clips with CMD+A, then render with CTRL+R. This avoids continuous rendering and build-up of discarded render files.

Please Log in or Create an account to join the conversation.

Last edit: by joema.

Waveforms very slow to rebuild 03 Nov 2020 11:31 #110680

Thank you Joe for the rundown of so much sound advice. I have been using 90% of those procedures you mention, so it's also good to know I am on the right path. It's very strange, for me, how it takes SO long to rebuild these waveforms. I have left FCPX alone for several hours, several sessions, and it still is rebuilding them. Every now and then a clip previously with red waveforms becomes complete. One by one, very slow, and I am on a not-so-slow 2019 Mac mini.
About Catalina and FCP 10.4.10, I di not know there was that update, will look into installing both on my 2017 MBP and see how they go. I would love to upgrade fully, but mid project and with 32-bit apps that I still like to use, I will not upgrade yet on my main machine (Mac mini).
Thank you!

Please Log in or Create an account to join the conversation.

Waveforms very slow to rebuild 03 Nov 2020 12:05 #110681

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1775
  • Karma: 27
  • Thank you received: 395

Alan Langdon wrote: ....I would love to upgrade fully, but mid project and with 32-bit apps that I still like to use, I will not upgrade yet on my main machine (Mac mini)....


You do not need Catalina, you mainly need FCPX 10.4.10 which runs fine on Mojave and has no problems with 32-bit apps. As with any FCPX update there are possible issues with plugins. Normally if you upgrade all plugins and get rid of old non-use plugins, it works fairly well.

If you make backup copies of your libraries and use Finder to duplicate and save the 10.4.6 Final Cut Pro.app bundle in /Movies, you can roll back to that within minutes.

The waveforms and thumbnails incur lots of small random IOs to the FCPX cache. It's important the cache be placed on a separate drive, ideally a fast SSD. That by itself will greatly improve performance. You also want a "lean" library with all media external. The library also should be on an SSD, possibly the same one as the FCPX cache. I would format all external drives HFS+, whether they are mechanical or SSD.

The Mac Mini has no discrete GPU, so that can be a significant issue for effects. However it should not cause slow waveform generation.

Please Log in or Create an account to join the conversation.

Waveforms very slow to rebuild 03 Nov 2020 13:24 #110686

Great to hear about 10.4.10 being OK with Mojave. Thanks, I'll consider upgrading after this project is over.
About your other tips, great ones again, some I already use (most, really)... but the HFS+ formatting of drives surprised me... Why not AFPS? Or even good old Mac OS Extended?
I currently have my Libraries on my internal APFS SSD, and all Media and Cache on an external Samsung T5 Ssd (not the fastest, but way better than my Lacie Thunderbolt 2 drive, I feel).

Please Log in or Create an account to join the conversation.

Waveforms very slow to rebuild 03 Nov 2020 14:26 #110693

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1775
  • Karma: 27
  • Thank you received: 395
Normally it shouldn't take hours to generate waveforms. On my iMac Pro running 10.4.10 with media on a 4-drive OWC Thunderbay 4 in RAID-0 and the library/cache on the system SSD, I imported 466 .wav files which totaled 273 GB. This took about 15 min to complete waveform generation.

I then did another test of importing 686 .mp3 files totaling 16 GB. They were scattered throughout a 48TB OWC array; I located them by doing a Finder search on *.mp3, selected them then imported via drag/drop with FCPX set to "leave files in place". That went through three phases:

(1) Validating 686 .mp3 files for import (5 min)
(2) Processing files for import (10 min)
(3) Waveform generation (22 min)
Total time from import start to completing waveform generation: 37.5 min.

Observing Activity Monitor's IO tab during phase #1 and #2, it was doing a very high read rate of over 6,000 per sec. These were certainly cached reads using the MacOS "Unified Buffer Cache" which uses surplus RAM to cache disk I/O.

In my .wav import test, the average file size was quite large. During the .mp3 import, the average file size was smaller. In Activity Monitor's IO tab if you compare read in/sec vs data read/sec, this gives a rough approximation of IO size. Likewise for writes. In general it appears FCPX does lots of small reads, then transitions to a phase where it's doing lots of small writes, likely writing waveform and peak data to the FCPX cache.

If your library, FCPX cache and media are all on the same mechanical disk, those IO streams may interfere with each other, slowing the process. This is similar to copying multiple large files between two mechanical drives using Finder. It is normally faster to copy them sequentially vs in parallel.

During waveform generation, if the Event Browser is set to filmstrip mode, it will prioritize non-generated waveforms on screen. If you keep scrolling down during that process, it might conflict with background waveform generation. To avoid this consider putting the Event Browser in "list view" while waveform generation is happening. OPT+CMD+2 toggles between filmstrip mode and list view.

In general if you have a large production using thousands of clips, you want higher-end hardware. You don't need a $40k Mac Pro but I'm unsure if a Mac Mini with all media on a single mechanical drive is adequate. At a minimum you want the cache and a lean library on an external SSD, preferably something like this Helix 2TB which can do 1,000 MB/sec and doesn't have thermal slowdowns under sustained writing: www.amazon.com/dp/B07YCR1S3K/

After the media is ingested and organized, and after initial editing is well underway, then you may start applying effects. Since those are GPU-based, the lack of a discrete GPU might be a problem. In theory you can add an eGPU but those don't work as well as an internal GPU.

Please Log in or Create an account to join the conversation.

Waveforms very slow to rebuild 03 Nov 2020 14:45 #110697

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1775
  • Karma: 27
  • Thank you received: 395

Alan Langdon wrote: ... but the HFS+ formatting of drives surprised me... Why not AFPS? Or even good old Mac OS Extended? I currently have my Libraries on my internal APFS SSD, and all Media and Cache on an external Samsung T5 Ssd (not the fastest, but way better than my Lacie Thunderbolt 2 drive, I feel).


HFS+ is just another name for Mac OS Extended Journaled. APFS is OK for external SSDs but it's no faster and you have no option for 3rd-party file system utilities. E.g, if data recovery is needed you cannot use Disk Warrior, etc. If it's already APFS I'd leave it that way.

I definitely would not use APFS on a mechanical drive, due to poor file enumeration performance: bombich.com/blog/2019/09/12/analysis-apf...tational-hard-drives

Somehow I was under the impression your media was on a mechanical drive. You are correct the T5 while not perfect is better than a rotating drive.

It appears you are already doing most of the best steps. If possible I'd suggest a careful, well-planned upgrade to 10.4.10 while staying on Mojave. Make sure all libraries and previous version of Final Cut Pro.app are backed up. After the upgrade do progressive tests to check for problems. 10.4.10 is significantly faster than 10.4.6 in several areas. Whether that would make a major difference for your "slow waveform" issue, I don't know.

Over the years there have been several performance improvements to waveform generation, but I don't recollect any since 10.4.5, at least officially-stated ones. But the other upgrades since 10.4.6 including Metal optimization of GPU functions and decode/encode performance probably warrant a serious look at upgrading.

Please Log in or Create an account to join the conversation.

Waveforms very slow to rebuild 03 Nov 2020 15:49 #110699

Thanks for so many detailed answers and tests on your end. I think you might have oversighted some of my answers, though: I have the Library on the internal SSD, Cache on external USB-C SSD (Samsung T5), Prxies on the same drive, and original media on a Lacie 7200 rpm which has Thunderbolt connecting via USB-C adapter.
As for effects and such, none yet, and since I have experience with effects and color correction, I am well aware to leave those for last, but thanks for the advice!
I know, my Mac mini is not Pro, but this is the third or fourth feature doc which I have edited on this or worse macminis, including heavy post production, and it always has worked: I live in a third world country where we earn small bucks so I have to make do with what I can purchase... Wish I could buy better equipment, for sure, but this setup I have is my best so far and has been great for my needs. An External GPU would be my wish list item, I guess, for DaVinci, etc...
Many thanks, again, you are being very helpful!

Please Log in or Create an account to join the conversation.

Waveforms very slow to rebuild 03 Nov 2020 15:50 #110700

Again, many many thanks! You have been giving me classes that I didn't even know I needed! MUITO Obrigado, as we say here in Brazil!

Please Log in or Create an account to join the conversation.

Waveforms very slow to rebuild 03 Nov 2020 16:03 #110701

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1775
  • Karma: 27
  • Thank you received: 395

Alan Langdon wrote: ... I think you might have oversighted some of my answers, though...


Understood -- you have already taken the best steps possible in current circumstances. If possible upgraded to 10.4.10 on Mojave, but as always there is some risk if in mid project. If all libraries and FCPX .app files are backed up, it should be low risk at least to do some tests.

In case you have some plugins or other software causing a memory leak, that can gradually slow performance. You can check for that by using Activity Monitor>Memory tab, and the memory pressure graph should be green. If yellow or red, there is a problem which should be pursued.

Please Log in or Create an account to join the conversation.

Last edit: by joema.
  • Page:
  • 1