Welcome, Guest
Username: Password: Remember me

TOPIC: FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely?

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 16 Oct 2019 05:57 #102074

  • tj7
  • tj7's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 107
  • Karma: 0
FCPX 10.3.4
OS Sierra 10.12.6'iMac late 2014, 27" i7, 32GB DDR3 RAM, AMD Radeon R9 M295X
Thunderbolt connected external HD

Probably a really dumb question:

How do you tell FCPX to select ALL media in a project file ie not just what's in the timeline but what has been loaded into the browser - and duplicate all of that media into a dedicated separate place, so you can clean up the source media HD location without accidentally deleting used media?

I am asking this strange question because the project I am working on - and have been for 3 years - is essentially a massive multimedia collage of hundreds of archival video clips, images and audio. So over the years, my source media folder from which everything is loaded has become absolutely huge - probably only about 25% its contents now exist in the library for this project, however, as I have at least well cleaned up the library within FCPX. I need to free up space on the source drive and would rather not select and delete files by hand one at a time after cross referencing manually - surely that's not the only way to properly clean up and minimise the source media folder so that it only contains the media files in the actual library.

So, put another way in case I am not clear enough:
I want to keep everything as is in my FCPX library / project / event etc. No changes. It's tidy and contains what should be there. All of this media is kept in its original folder, so in my "original media" folder within my FCPX library package on my HD, there are aliases to all of these files only. I want to clean up the source media HD from which everything has been loaded into FCPX and delete the media which is not loaded into my FCPX library. So I wonder if there is a way to either duplicate the actual media which is in my FCPX library elsewhere, allowing me to delete the source media HD and then refill it with the copied media so it's cleaned up - or is there another way of achieving the same aim?

I really hope this makes sense -

Thanks!
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 16 Oct 2019 12:07 #102075

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1265
  • Thank you received: 263
  • Karma: 22
One way is simply consolidate the library to another storage location. That way only currently-imported material will be moved. The downside is during consolidation the file/folder structure will be changed. You can get an idea of what those changes might be by creating a new library on the new storage location, and copying one event across from old to new library, consolidating, then examining the new folder structure in Finder. Since with FCPX you normally search your media via powerful database tools inside FCPX, this may not be an issue.

Another way is use Final Cut Library Manager to make a .csv file of all existing media items in your library, then move those manually via Finder or other file-level tools to the new location, then relink. That is often less work than first appears because they are usually in a few folders. www.arcticwhiteness.com/finalcutlibrarymanager/

However if you have a complex media folder tree with only some items imported "in place", then you'd need to take the .csv file, maybe export to .txt and use rsync, awk or some other tool to read that copy each file/folder to the new location, then relink in FCPX.

If you have a mix of "copied to library" and "in place" media, I think the only solution is define a new storage location and consolidate. You can determine this by using Final Cut Library Manager, making a .csv media list and looking at the "Location" column which states whether each media file is internal or external.
Last Edit: 16 Oct 2019 12:11 by joema.
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 16 Oct 2019 13:43 #102076

  • Francois
  • Francois's Avatar
  • OFFLINE
  • Gold Boarder
  • Posts: 228
  • Thank you received: 38
  • Karma: 4
I would also check if this software REFERENCE X could be useful in your situation ...
www.fcp.co/final-cut-pro/news/2038-copy-...o-x-with-reference-x
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 16 Oct 2019 14:10 #102078

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1265
  • Thank you received: 263
  • Karma: 22
Francois, thanks. I forgot about ReferenceX. That is exactly what tj7 needs.

You simply make a library xml file, import to ReferenceX, then press the copy button, specify the destination in the dialog, and it copies only the media files used in the library to a new location. They are copied not moved.

It copies both internal media imported to the library plus external media imported "in place". So the new destination will contain all the media used or referenced by the library. After the copy the files in that new destination can be relinked.

You can also do this for only a single project. For each copy operation ReferenceX provides a .csv log which states what files were copied and whether they are referenced by a project.

This is a great tool, simple to use and very useful.
Last Edit: 16 Oct 2019 14:29 by joema.
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 16 Oct 2019 23:41 #102083

  • tj7
  • tj7's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 107
  • Karma: 0
Wow - ReferenceX sounds exactly right. Thanks!
What a smart little app - what you've described is precisely what I was asking for, so I will try this out!

"It copies both internal media imported to the library plus external media imported "in place"
This would mean that I can complete this task, then delete the original HD storage location and copy back only the media which ReferenceX has copied for me, thus achieving my cleanup. I'll back it all up first though, to be safe of course.
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 17 Oct 2019 12:31 #102088

  • thinkfilm
  • thinkfilm's Avatar
  • OFFLINE
  • Gold Boarder
  • Posts: 168
  • Thank you received: 25
  • Karma: 5
Just one note: I see tj7 that you are still on Sierra so it may not be an issue but at least one other user has reported that Reference X crashes on OSX Catalina. I've pinged the developer (Tokyocode.io) and asked if there is an update (coming).
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 17 Oct 2019 22:59 #102100

  • tj7
  • tj7's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 107
  • Karma: 0
thinkfilm wrote:
Just one note: I see tj7 that you are still on Sierra so it may not be an issue but at least one other user has reported that Reference X crashes on OSX Catalina. I've pinged the developer (Tokyocode.io) and asked if there is an update (coming).

Yes I am stuck on Sierra for some time, until I can afford to upgrade Pro Tools!
Well, I just hope the version of ReferenceX which I buy is ok with an older OS like Sierra.
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 19 Oct 2019 00:45 #102125

  • tj7
  • tj7's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 107
  • Karma: 0
ReferenceX has proven to be dicky and creating more stress: It hangs at varying points during copy every time.
Have spent 45 mins with it now, babysitting it and then force quitting and restarting the process. Have contacted the developer, will see what emerges...

Post - ok, so I "completed" the process after many hangs, quits and restarts of the app. However I have no idea if there are now hundreds of corrupted, partially copied files or not. The app itself uses a colour code system for file names and "copy status" but doesn't tell the user what these colours mean. There is no manual. So now I have a bunch of green AND yellow "copy status" dots here and about half of my file names are yellow vs grey. I've tweeted all to the developer, hopefully they can just fwd me an older version which works fine under Sierra.
Last Edit: 19 Oct 2019 01:58 by tj7.
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 19 Oct 2019 14:04 #102134

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1265
  • Thank you received: 263
  • Karma: 22
tj7 wrote:
ReferenceX has proven to be dicky and creating more stress: It hangs at varying points during copy every time.
Have spent 45 mins with it now, babysitting it and then force quitting and restarting the process. Have contacted the developer, will see what emerges...

tj7, sorry about the problems you are encountering. I just tested ReferenceX on various scenarios and I've also seen some problems, even on Mojave 10.14.6 and FCPX 10.4.7. I have tried fcpxml versions 1.8 and 1.7.

It works OK on small test cases but for larger libraries ReferenceX will often hang during the fcpxml analysis phase.

If it completes analysis and I start the copy, it will just sometimes quit without any error. I don't have yellow dots, just green ones where it finished and blank dots where it didn't copy.

If this utility worked reliably it would be very nice. Anytime an app handles large amounts of important data it is incumbent on the developer to thoroughly test this, but that's easier said than done. I will contact the developer and see if I can help.
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 19 Oct 2019 22:20 #102142

  • tj7
  • tj7's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 107
  • Karma: 0
Is there any way to check file integrity for multiple files at once?
I have a "copy" of all files used, but a blend of coloured dots and so far the only terminal commands I can find are for comparing one folder against another (no good as obviously the source folder contains a lot more files) or one by one file analysis...
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 19 Oct 2019 22:28 #102144

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1265
  • Thank you received: 263
  • Karma: 22
tj7 wrote:
Is there any way to check file integrity for multiple files at once?...

The best tool for this is Beyond Compare. It is absolutely superb and can do many different types of comparisons. It can even do "flat structure orphan verification", where it checks that a list of files on one drive all exist somewhere on another drive having a different folder structure.

www.scootersoftware.com

The current version is 32-bit but they have a 64-bit version in beta and will be out soon for Catalina.
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 19 Oct 2019 22:44 #102145

  • tj7
  • tj7's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 107
  • Karma: 0
Thanks - there's a free trial as well.
Very curious to see if this works for comparing
source A with source B - each containing the same file structure but A is 328GB of data and B is 88GB
(I told you I had a lot of cleanup to do!). Does it matter whether it's 32 or 64? That's probably a dumb question... it's something I only pay attention to for Pro Tools plugins....
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 20 Oct 2019 02:20 #102150

  • tj7
  • tj7's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 107
  • Karma: 0
Not having much luck here. It's been stuck on "saving report to file" for 20 minutes.
Happens every time I try to create a report to see how things went :(
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 20 Oct 2019 19:10 #102164

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1265
  • Thank you received: 263
  • Karma: 22
tj7 wrote:
Not having much luck here. It's been stuck on "saving report to file" for 20 minutes.
Happens every time I try to create a report to see how things went :(

You mean ReferenceX was stuck or Beyond Compare? ReferenceX definitely has some problems. I am still trying to contact the developer.

Beyond Compare is very reliable, however it has various comparison modes which trade off speed vs thoroughness. Often you can compare based on file name, file size, and timestamp, which is very fast.

Before doing large file comparisons it's usually best to de-duplicate the volume or folder tree. Otherwise duplicate files can throw off the comparison. There are many utilities for this but I use Gemini 2: macpaw.com/gemini
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 20 Oct 2019 22:12 #102168

  • tj7
  • tj7's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 107
  • Karma: 0
It's Beyond Compare. Forgive me but I'm sort of wary of installing a *third* application to do what the 2nd and first ones won't - feels like this is getting a little crazy...

I have had no response from the ReferenceX Dev - there is literally no manual or guide to explain what the colour codes for his dots mean.

I have posted the following at the BC forum and irritatingly they haven't approved the post - most likely takes a few days - so nobody has seen it:

4.3.0, build 24364, released Oct. 9, 2019
Mac OS Sierra 10.12.6

Have tried several times, summary / side by side
It just hangs on "Saving report to file"
Program is not "crashed" as force quit does not show BC as "unresponsive"

See attachment for stats of my comparison

Same problem with "copy report to clipboard"

Same problem even if I compare only one location containing only 30-40 files

App will not generate comparison summary at any point, even when I select option to generate during comparison

It hangs every single time


_________

(The upload attachment form here doesn't work, btw - I hit "add file" and then "insert", I see the code added to my post, then "submit", but no image - have tried chrome and safari several times...had to use imgbb.com)
Last Edit: 20 Oct 2019 22:22 by tj7.
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 20 Oct 2019 23:25 #102170

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1265
  • Thank you received: 263
  • Karma: 22
tj7 wrote:

Have tried several times, summary / side by side
It just hangs on "Saving report to file"
Program is not "crashed" as force quit does not show BC as "unresponsive"

See attachment for stats of my comparison

Same problem with "copy report to clipboard"

Same problem even if I compare only one location containing only 30-40 file....

Sorry about the problems. I just used the same version to compare two 20 TB folder trees each containing 23,000 files within 2,000 deeply-nested folders on two different 32 TB RAID-0 Thunderbolt arrays and it took about 30 sec for the initial compare then 12 sec for subsequent compares.

I saved the comparison result to both clipboard and to a text file in /Documents, and it worked fine.

If you look under the menu Session>Session Settings>Comparison, try setting it to just compare file size, and timestamps within 2 sec. Then at the bottom left of that dialog pick "Also update session defaults". That makes it persistent until the next time you change the comparison criteria.

You can add more stringent tests (if needed) and further adjust the exclude list once you get it working right.

Under the Name Filters> Exclude Files box, you can add some common "junk" files so it won't list or try to compare those. I use:

Exclude files:
.DS_Store
._*

Exclude folders:
_CCC SafetyNet
*.fcpcache
.Trashes
.Spotlight*
.fseventsd
._*

Since it integrates with Finder I usually right-click on the left side folder, in the Finder pop-up menu pick Services>Select Left Folder for Compare, then right-click on the right side folder to compare, Services>Compare Folders.

It will not compare two volume names at the highest level, you must have a folder name called "root" or something.

At the top the main buttons I use are Diffs (see dropdown choices), Same, and Structure (see dropdown choices).

One example: you have a smaller drive on the left and want to ensure you have those files somewhere within a much larger drive on the right having a different folder structure. You pick the left and right sides, then for Diffs pick "Show Left Orphans", then for Structure pick "Ignore Folder Structure". For each file on the left it will look through the entire huge drive on the right and verify whether each of those exist somewhere within it.

In general I suggest carefully de-duplicating a drive or folder tree before doing elaborate comparisons, especially for orphans.
Last Edit: 20 Oct 2019 23:28 by joema.
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 21 Oct 2019 00:26 #102171

  • tj7
  • tj7's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 107
  • Karma: 0
I changed prefs as suggested and made no difference.
It runs a comparison, I see it working and there is a black "=" sign in the centre column which appears for each file,
as per my screenshot in my last post. That might be all the indication I need?
However, this thing steadfastly refuses to spit out any kind of report. ANY kind.
It hangs on "Saving" forever. Same if I try the same task with two very small, simple folders to compare.

I'll have to see if I can "deduplicate" based on your advice, but so far each step seems to require new sub steps, each app seems to have one glitch or another, and I might have done better just to arduously do this by hand.... although I appreciate your help a lot!
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 21 Oct 2019 15:47 #102179

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1265
  • Thank you received: 263
  • Karma: 22
tj7 wrote:
I changed prefs as suggested and made no difference.
It runs a comparison, I see it working and there is a black "=" sign in the centre column which appears for each file,
as per my screenshot in my last post. That might be all the indication I need?
However, this thing steadfastly refuses to spit out any kind of report. ANY kind.

Sorry, I can't explain that. I have used Beyond Compare for years on Windows and later on many different Macs since 2013, I think that was on Sierra or High Sierra. It generally works very well.

tj7 wrote:
...I'll have to see if I can "deduplicate" based on your advice, but so far each step seems to require new sub steps, each app seems to have one glitch or another, and I might have done better just to arduously do this by hand.... although I appreciate your help a lot!

In the old days we captured a few DV tapes and there was little file management. Today we accumulate thousands of files, sometimes occupying hundreds of terabytes of drive space. There are constant issues with archiving, file management, space management, etc. I spend more of my time on IT-type issues than NLE issues. That's just the way it is. Anything you can learn about this area helps. Ideally a tech-savvy 1st Asst. Editor handles this area.

I know you are on Sierra for compatibility reasons with Pro Tools but the older the OS the more problems and incompatibilities can occur. E.g, many Mac users on back-level macOS versions were totally shut down due to the recent Chrome bug which corrupted their hard drives. They thought going to new macOS version entailed risk but that would have protected them (assuming they did not disable System Integrity Protection).

Upgrading major apps entails cost but you must also ask yourself what is your time worth to debug problems possibly related to old versions.
Last Edit: 21 Oct 2019 15:47 by joema.
The administrator has disabled public write access.

FCPX Source Media Folder Cleanup Q - How to clean up source media folder safely? 22 Oct 2019 02:53 #102182

  • tj7
  • tj7's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 107
  • Karma: 0
Essentially you make a fair point on the upgrade front.
It's just not as simple as that when apps such as Pro Tools cost in excess of AU$500 and you have a mortgage to be paying off. Personally, I find the endless insistent pressure to upgrade now, UPGRADE now. UPGRADE NOW, NOW! UPGRADE! NOOOWWWWW! to be mind-boggling. I have been perfectly happy with FCPX and Pro Tools versions under Sierra. I rarely need to install software. It is a climate of constant upgrades which results in the user experience being one of being perpetually "behind". Anyway - this is all neither here nor there, hopefully BC get back to me with a solution, or at least clarification that the black "=" symbol in the centre column actually indicates that each of those comparisons - all 2400+ of them - demonstrate that the copied files are all ok, even if the system won't generate a report.
The administrator has disabled public write access.