fbpx
Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1

TOPIC: How to move 2TB of Managed media into the External media location with 7TB

How to move 2TB of Managed media into the External media location with 7TB 02 Jun 2020 20:37 #107622

  • BJansch
  • BJansch's Avatar Topic Author
  • Offline
  • Senior Boarder
  • Senior Boarder
  • Posts: 73
  • Thank you received: 2
Hi, I'm currently finishing a feature documentary. FCPX has become extremely sluggish despite working on a Mac Pro and Raid 0 drive. My combined media is approx. 9TB of which nearly 2TB is managed media. I would like to make the Library fully External but I'm not sure how to go about moving those 2TB of Managed media into the External media folder.

I understand that the consolidate media command can assign an external folder, but FCPX wants to move ALL media into that folder including the already External media.

Any ideas best to do this. The goal is to be able to move projects to other Libraries without the need to copy media, hence the reason why I'm trying to move the Managed media to the External folder.

Sorry, probably a long winded way to describe it...

My gear: Mac Pro, 12 Core, FCPX latest version,Catalina 15.something

Thanks for any tip!

Boris

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 02 Jun 2020 21:06 #107623

  • Tom Wolsky
  • Tom Wolsky's Avatar
  • Offline
  • Moderator
  • Moderator
  • Posts: 4480
  • Karma: 112
  • Thank you received: 733
It doesn’t move external media. It uses symlinks.

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 02 Jun 2020 23:45 #107625

  • BJansch
  • BJansch's Avatar Topic Author
  • Offline
  • Senior Boarder
  • Senior Boarder
  • Posts: 73
  • Thank you received: 2
I'm not trying to move external media, I'm trying to move managed media to the external media folder ie consolidate all media into that location

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 03 Jun 2020 00:15 #107626

  • Tom Wolsky
  • Tom Wolsky's Avatar
  • Offline
  • Moderator
  • Moderator
  • Posts: 4480
  • Karma: 112
  • Thank you received: 733
The managed media is moved to the designated folder on the external drive. The external media will have representations for it created in the designated location. I’m not seeing what’s confusing you about the consolidate process.

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 03 Jun 2020 09:43 #107631

  • BJansch
  • BJansch's Avatar Topic Author
  • Offline
  • Senior Boarder
  • Senior Boarder
  • Posts: 73
  • Thank you received: 2
Thanks Tom, so if I create a new folder within the External media folder and tell FCPX to consolidate media into that folder, it won't duplicate any of the external media, just create shortcuts?

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 03 Jun 2020 12:05 #107634

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1516
  • Karma: 27
  • Thank you received: 329

BJansch wrote: ... if I create a new folder within the External media folder and tell FCPX to consolidate media into that folder, it won't duplicate any of the external media, just create shortcuts?

Consolidation will duplicate external media which was imported with "leave files in place", but (depending on the disk config) hard link optimization might avoid additional space consumption. Let me re-state & you make any needed corrections. You have 9TB external media imported with "leave files in place", and 2TB managed media, probably inside the library.

You don't want the split managed/unmanaged storage but want it all external, BUT you don't want to pay the duplicate storage penalty of consolidating the external "in place" media to a common location with the managed media.

Unfortunately you cannot selectively move *only* internal managed media to an external "in place" location - at least using the FCPX UI. Also if dealing with hundreds/thousands of files in many folders, the FCPX UI doesn't make it easy to see file pathname info.

The only official procedure is designate a new storage location in FCPX using the Library Inspector>Storage Locations>Modify Settings>Media, then consolidate. This MOVES *all* internal managed media *and* COPIES external "in place" media to that new location. Thus it superficially duplicates the external unmanaged media. Also it does not maintain the external media folder structure, which in turn creates possible filename collisions, which FCPX handles by renaming the media files.

Fortunately if all media is on a single HFS+ disk volume, FCPX will use the HFS+ "hard link" optimization during the consolidation. After the consolidation you may see both original "in place" media files plus a copy of those in the new designated location, but these are actually hard linked to a single physical file. Thus no additional space is required. Once you verify the consolidation is successful you can delete the original external media.

This can be verified using the terminal commands ls -l which displays the file's link count and ls -i which displays the file's inode number. After the consolidation the link count will increase by 1, which indicates two apparent files are hard linked to a single physical file. ls -i will show both files have the same inode, thus are the same actual file, taking up no additional space. This doesn't work on ExFAT, and I don't know how NAS or APFS handles it.

There is a more selective procedure but it involves going inside the library and manually moving files. You can go inside the library bundle and in /Event Name/Original Media, use Finder to move (not copy) those files to an external storage location. You must be careful because there will be a mix of symlinks to "in place" media plus the managed files themselves. You only want to move the managed files. Then restart FCPX, those now-missing files will be red clips, then you just relink them. Note this only works for regular media files, not proxies.

To help see where all the media files are, you can use the 3rd-party utility Final Cut Library Manager, which has an add-on feature to make a CSV list of all file pathnames used in a library: www.arcticwhiteness.com/finalcutlibrarymanager/

Please Log in or Create an account to join the conversation.

Last edit: by joema.

How to move 2TB of Managed media into the External media location with 7TB 03 Jun 2020 12:27 #107635

  • BJansch
  • BJansch's Avatar Topic Author
  • Offline
  • Senior Boarder
  • Senior Boarder
  • Posts: 73
  • Thank you received: 2
Thanks v much Joema, that's what I call an answer!

I like the manual, move managed media option, from the Original media folder to a new folder within the External unmanaged med folder and then relinking. The alternative sounds too scary to me and I don't have enough space to duplicate 9TB on the one drive, which is also APFS, so that might also be an unknown factor...

Great advice and much appreciated!!

Best,
Boris

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 03 Jun 2020 12:46 #107636

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1516
  • Karma: 27
  • Thank you received: 329

BJansch wrote: ... I don't have enough space to duplicate 9TB on the one drive, which is also APFS, so that might also be an unknown factor...


I think the APFS "copy on write" feature provides similar functionality to HFS+ hard links but I have not tested this.

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 03 Jun 2020 13:09 #107638

  • BJansch
  • BJansch's Avatar Topic Author
  • Offline
  • Senior Boarder
  • Senior Boarder
  • Posts: 73
  • Thank you received: 2
Would you mind me just asking you what you would do in my situation?

I have a full length feature doc. FCPX is slow as FCUK, just adjusting audio levels causes it to think for 3-4 seconds. It's a new Lacie Big 16TB RAID0 enclosure. Half the footage is proves 422HQ, the rest .h265, which play perfectly, but while playing the probes the audio cuts out randomly, even when the picture is rendered.

My Mac Pro is up to spec, 128GB RAM, 12 Cores, decent gfx card (not the best but the upgraded W7000 or something)

So my thinking was I'll copy the master project over to a new library for tweaking at speed, but now I have this Managed media problem...

Any further tips hugely appreciated!

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 03 Jun 2020 13:14 #107639

  • Tom Wolsky
  • Tom Wolsky's Avatar
  • Offline
  • Moderator
  • Moderator
  • Posts: 4480
  • Karma: 112
  • Thank you received: 733
Consolidate your media into a folder on the same drive as your other media.

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 03 Jun 2020 13:38 #107640

  • BJansch
  • BJansch's Avatar Topic Author
  • Offline
  • Senior Boarder
  • Senior Boarder
  • Posts: 73
  • Thank you received: 2
Thanks Tom, my wife is a journalist so I do like terse, but if my goal is to speed up FCPX and my 9TB project, is that going to help do you think? As I said to Joema, FCPX hangs for 3-4 seconds when I do something as simple as adjusting audio level on a music track and even rendered prores 422HQ has audio dropout on my brand new arse kicking Mac Pro!?! The film is 95% complete, so I just want a timeline I can work with without waiting around while it thinks about how big the library is... hmmm

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 03 Jun 2020 13:47 #107642

  • Tom Wolsky
  • Tom Wolsky's Avatar
  • Offline
  • Moderator
  • Moderator
  • Posts: 4480
  • Karma: 112
  • Thank you received: 733
I wouldn’t think consolidating would improve your performance.

Please Log in or Create an account to join the conversation.

How to move 2TB of Managed media into the External media location with 7TB 03 Jun 2020 13:49 #107643

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1516
  • Karma: 27
  • Thank you received: 329

BJansch wrote: ...I have a full length feature doc. FCPX is slow as FCUK, just adjusting audio levels causes it to think for 3-4 seconds. It's a new Lacie Big 16TB RAID0 enclosure. Half the footage is proves 422HQ, the rest .h265, which play perfectly, but while playing the probes the audio cuts out randomly, even when the picture is rendered...


If the timeline is fully rendered it should never be slow or have 3-4 sec lag when adjusting audio. Adjusting clip audio levels does not cause a re-render. Verify you are not running Chrome or any browser based on Chrome. That app is poorly behaved and can saturate the MacOS VideoToolBox framework, causing random and intermittent FCPX hangs.

Unless you have a 2019 Mac Pro you generally don't have hardware acceleration for Long GOP formats like HEVC/H.265. You could create a smaller test project and transcode the HEVC to optimized media; see if that makes a difference.

Starting with FCPX 10.4.7 and continuing in 10.4.8 there is a narrow issue with certain HEVC/H.265 codecs that can cause an FCPX hang. I don't have a comprehensive list but I've seen it on material from a DJI Inspire 2, Arlo security camera, and it's been reported on a Panasonic EVA-1. For these reasons consider making a smaller test timeline and transcoding any HEVC to optimized media. If it ever hangs during the transcode, that also signifies the problem.

If you have imported "in place" any bare MTS files from AVCHD media, that can poison the I/O performance of the entire library. Make sure you have none of those. If so transcode them to optimized media.

If you have any external audio or video devices, temporarily disconnect those and see if it makes a difference.

As a standard troubleshooting procedure, make sure background rendering is disabled. Delete all cache files with Files>Delete Generated Library Files>Delete Render Files>All. Then do a one-time render of the timeline with CMD+A to select and CTRL+R to render. This avoids constant rendering, possible render tracking issues and reduces space consumption from render files which are never deleted.

Multiple ProRes layers can cause high I/O (if not rendered). But if the timeline is rendered, all layers are computed and flattened to a single set of render files.

If the timeline is truly rendered there should also be no playback overhead from effects. However there are a few cases where the render tracking system malfunctions and it will re-compute the effect even though no render dots are showing. This could theoretically also increase I/O, causing FCPX to fetch data from multiple ProRes files vs the render cache.

For the above reasons it's probably a good idea to investigate what Fx are being used. Certain Fx can negatively interact with the render system. The current Fx Plug 3 framework does not give developers control over when their Fx are rendered. In some cases FCPX will yank control and start rendering. This is especially apparent with compute-intensive Fx like Neat Video.

You could create a snapshot duplicate of your current timeline, strip all Fx with Edit>Remove Effects, then examine the performance. If it becomes consistently fast, that indicates a possible Fx problem or interaction with the render system. In that case duplicate it again but only remove Fx from 1/2 of the timeline, then 1/4, etc to narrow down what region and what Fx is causing the problem.

Re whether it's an I/O constraint or computational issue, ideally Activity Monitor would have parameters for disk queue depth (like Windows PerfMon) but unfortunately it doesn't. At least when it's in a 4-sec delay, you can watch Activity Monitor for CPU vs GPU vs I/O activity. Using the 3rd-party utility iStat Menus makes this easier to monitor. You can also differentiate between slow app behavior and a beach ball. The beach ball indicates the MacOS window manager noticed the app is not responding to events. Normally this should not happen except very briefly and infrequently. If it happens significantly something in the app is getting stuck such as an Fx or an I/O to an external device is hanging. Thus the suggestion to temporarily disconnect any external video or audio hardware.

Please Log in or Create an account to join the conversation.

Last edit: by joema.
  • Page:
  • 1