fbpx
Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1

TOPIC: media asset management system

media asset management system 30 Jul 2020 19:43 #109042

  • Glitchdog
  • Glitchdog's Avatar Topic Author
  • Offline
  • Fresh Boarder
  • Fresh Boarder
  • Posts: 3
  • Thank you received: 0
We are finally back using FCPX, yay!! We started with Avid to Media 100, then FCP2-7, then Premiere. Happy to be here and on Final Cut again.

We are in the market, as a budget non-profit, for a media asset management system. As I look more at the rich metadata within Final Cut, I don't see much out there so far that has a really good ability to read/write FCPX metadata.

Our setup is currently one editor with a second one in place in the next year or two. We want to work off our internal Windows server currently (a MacMini server in-between is fine), then eventually migrate to the cloud, once pricing has dropped to fit our budget. Would like a system that doesn't nickel and dime you to death. Wondering what others are using that you feel is a good marriage between FCPX and the MAM?

Please Log in or Create an account to join the conversation.

media asset management system 31 Jul 2020 13:43 #109052

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1551
  • Karma: 27
  • Thank you received: 331

Glitchdog wrote: ... budget non-profit, for a media asset management system. As I look more at the rich metadata within Final Cut, I don't see much out there so far that has a really good ability to read/write FCPX metadata.

Our setup is currently one editor with a second one in place in the next year or two. We want to work off our internal Windows server currently (a MacMini server in-between is fine), then eventually migrate to the cloud, once pricing has dropped to fit our budget. Would like a system that doesn't nickel and dime you to death. Wondering what others are using that you feel is a good marriage between FCPX and the MAM?


There are now FCPX workflow extensions that integrate and provide MAM capabilities. However you don't necessarily need these and can "roll your own" collaborative system, but at a possibly significant cost in investigative and testing time.

Frame.io: frame.io
KeyFlow Pro 2: www.keyflowpro.com

See also PostLab: hedge.video/postlab/concepts

FCPX by itself is really good at data organization but scope is limited to a single library. What you'd ideally like is cross-library searching. I think Frame.io or KeyFlow Pro give that, but you can do it yourself manually with the inexpensive tool FindrCat. Using this method you'd curate, rate and keyword imported media within a given FCPX library, then export using FindrCat which preserves those as Finder tags on the media files, making them searchable using the MacOS Spotlight indexing system.

Data organization using the FCPX rating and keywording system is heavily range-based, not just clip-based. I think FindrCat and Spotlight indexing are all clip-based. Ideally you'd like a cross-library MAM workflow extension which preserved FCPX's range-based tagging. I don't remember if Frame.io or KeyFlow Pro do that.

Theoretically you save FCPX keywords and ratings across multiple libraries using FindrCat, but it would assume those datasets were always on line. It's probably possible to subsequently index those with NeoFinder, producing a searchable database of offline media: cdfinder.de/guide/8/8.7/neofinder_tags.html

Note Spotlight indexing may have limitations on a NAS drive and is not designed for intermittently-connected cloud assets.

Please Log in or Create an account to join the conversation.

media asset management system 31 Jul 2020 15:12 #109055

  • Glitchdog
  • Glitchdog's Avatar Topic Author
  • Offline
  • Fresh Boarder
  • Fresh Boarder
  • Posts: 3
  • Thank you received: 0
Thanks for the info Joema. I stumbled on FindrCat Pro several weeks ago and wondered about implementing that regardless of what we do. If I understand correctly the program actually puts all the FCPX metadata as Finder tags so it's directly connected to each clip of media. My limited understanding with MAMs is that they use some kind of XML sidecar, so the metadata is not often readable outside their dedicated program.

One clarification. All our new footage gets used for the particular project at hand, but then also reused over and over for other projects. I'm still using HD footage from 10 years back in some projects. So easily tracking all this and knowing what shots have been used in which projects, so we don't reuse for the same audience too frequently. Also being able to have searchable access to all the footage as proxies easily from home or office, but then pull down full-res clips I need to the RAID, etc.

I did look at both KeyFlow Pro and Kyno (though trying to get away from subscription model), both very cool programs from what I can tell at a surface glance. But I'm not sure if they would be the best choice long-term. Still, I'm waiting to hear back from KeyFlow with my list of questions along with a software app, Cantemo I discovered on this site.

What I was hoping to find is a MAM that can sync with all the metadata in FCPX. BUT since I've just started wrapping my head around the FCPX metadata, I wasn't sure if I really needed something (if it exists) that can sync with all the FCPX metadata. Do I need all the FCPX metadata accessible in an asset management setup?

Please Log in or Create an account to join the conversation.

media asset management system 01 Aug 2020 11:41 #109061

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1551
  • Karma: 27
  • Thank you received: 331

Glitchdog wrote: ...What I was hoping to find is a MAM that can sync with all the metadata in FCPX. BUT since I've just started wrapping my head around the FCPX metadata, I wasn't sure if I really needed something (if it exists) that can sync with all the FCPX metadata. Do I need all the FCPX metadata accessible in an asset management setup?


FCPX metadata is stored within various SQL tables within several SQLite databases within a FCPX library. That info is undocumented and in a format that can be humanly understood. Normally with a SQL database a human devises the schema - table names, column names, datatypes, etc, and also writes the queries. However internally FCPX uses Apple's "Core Data" framework, where the programmer-facing data view is an object graph, and Core Data itself creates and uses the underlying SQL data store. So while the tables can be queried with a 3rd party tool, the info is not understandable.

This leaves the only method of getting metadata info as XML. The application-specific details are also not documented but it's possible to figure some things out, so some 3rd parties have written tools for that, such as MergeX which is now owned by PostLab: www.merge.software

With any long-duration MAM approach (whether primitive and manual or a sophisticated 3rd-party database), durability and support are major issues. E.g, say you spend thousands of hours tagging photos using LightRoom. In 8 years will that database be available, readable and intact? Does the app developer allow exporting the data (with all features) in a standards-based format such as CSV or something else in case their database crashes or goes non-supported?

CatDV has been around a long time, now has an FCPX workflow extension, is widely used in the industry and has the features you want: www.squarebox.com/fcpx/

However it is likely expensive. Also (even given the funding) you can't just pick a product and commit. What if under high stress or heavy load it became unreliable or damaged data? What if that only happened using a certain client OS or only on a NAS if using a certain network protocol? Who would support and debug that? These issues apply to any 3rd-party MAM.

Even using the relatively simple Mac Spotlight Indexing, all features of this do not always work consistently on all NAS platforms. IOW you could spend lots of time using Finder tags on a local drive, move it to the NAS, then find an indexed search of those tags doesn't consistently.

For these reasons some people fall back and use the simple approach of embedding metadata in filenames. It is painfully primitive but at least filenames are durable. E.g, camera file C00001.mp4 becomes 2018_ATL_Wedding_Smith_Reception.mp4. Whether a database or rudimentary filenames you must decide on a consistent naming convention convention ahead of time then stick to that -- it can't be "stream of consciousness.

However the above filename approach can conflict with FCPX reliance on filenames at ingest time, esp if initial curation is done within FCPX. That raises yet another issue: traditionally initial curation took place outside the NLE because (a) NLE tools for tagging data were poor or (b) The assistant editor didn't want "all that junk in my library".

With FCPX it's often faster to just ingest everything using "leave files in place" and curate with the skimmer, ratings and keywords. No external tool is that fast, not even Kyno: lesspain.software/kyno/

But once ingested FCPX is reliant on the filename and if later changed this breaks the link (if not on the original drive where "inode lookup" is possible). This in turn argues for deciding for all time on a filename convention *before* ingest. There is no good answer for this issue.

That said, I recommend you investigate closely Kyno. It is non-subscription, cross-platform and may fit your needs:

lesspain.software/kyno/

Please Log in or Create an account to join the conversation.

media asset management system 01 Aug 2020 23:48 #109070

  • cofe
  • cofe's Avatar
  • Offline
  • Expert Boarder
  • Expert Boarder
  • Posts: 133
  • Karma: 2
  • Thank you received: 26
Joema inside is always amazing and some fantastic advice in there.

One though that crossed my mind when reading the thread: Why not FCPx as your MAM? Depending on your project sizes you can make libraries self contained (store media within the container).
I'm doing this on a current project currently (works well for my 6 min stories, might not if you do several feature length docs). I store the original media externally, but keep the FCPx proxies within the library container. I edit in proxy mode and when I need footage from another project I just open that Lib and FCPx will pull the enquired proxy into the current Lib when I edit the clip into a timeline. When I need to online I either just connect the online drive(s) or copy it into the current projects external online media location via the consolidate command.
You could even create custom Libs that hold all those generic clips you encounter over time for quicker access.
This system needs a little planning and discipline but you avoid another software and all the potential hassles Joema listed. It also might be nicely assisted with PostLab, even more so with a proxy workflow in the cloud via their upcoming 'Drive' feature.

Anyway just a thought

Please Log in or Create an account to join the conversation.

media asset management system 02 Aug 2020 13:48 #109082

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1551
  • Karma: 27
  • Thank you received: 331

cofe wrote: ... Why not FCPx as your MAM? Depending on your project sizes you can make libraries self contained (store media within the container)...I store the original media externally, but keep the FCPx proxies within the library container. I edit in proxy mode and when I need footage from another project I just open that Lib and FCPx will pull the enquired proxy into the current Lib when I edit the clip into a timeline. When I need to online I either just connect the online drive(s) or copy it into the current projects external online media location via the consolidate command.

You could even create custom Libs that hold all those generic clips you encounter over time for quicker access.
This system needs a little planning and discipline but you avoid another software and all the potential hassles Joema listed. It also might be nicely assisted with PostLab, even more so with a proxy workflow in the cloud via their upcoming 'Drive' feature.


Yes that can work. A few possible issues: Each major FCPX upgrade requires library upgrade. In general this is reliable but if you had a 4-yr-old archival library then opened it with the latest FCPX that would be a big upgrade. That problem is not unique to FCPX -- any database content system will periodically require database upgrades.

There is a soft upper limit on the current FCPX library database. I have used libraries with 8,500 clips and 200+ camera hours of material and it worked OK but the headroom is not infinite. Each event is implemented as a separate SQLite database so splitting library across events may help, but there is no need to over-do it. A single event can manage a huge amount of content.

There is another soft limit based on # of projects or snapshots. Each project also is implemented as a separate SQLite database, and each open db entails system and memory overhead. FCPX uses a "deferred opening" algorithm to try and avoid opening an excessive number of projects or events but sometimes with over 30-50 projects it will slow down.

In general these limits are very high (except the # of projects) and even a huge feature film (inc'l all footage from all cameras) can be managed in a single library, given appropriate hardware. But there's a difference between that and a long-term archival library which might contain clips from multiple films or documentaries. The main constraint is each library is absolutely independent and FCPX cannot query across libraries.

Proxies if stored externally can be somewhat fragile. The original proxy design was in-library only and that is more reliable but it bloats library size. For longer-term archival storage I recommend also grabbing the FCPX auto-backups for a given library which are stored in /Movies/Final Cut Backups. These are very small and contain no proxies, cache or even symlinks. If after years you try to open or upgrade a regular library, having these around might be useful. I would also suggest exporting library and project XMLs as part of the archive package.

Re cross-library copying of projects and clips, there are some FCPX media management issues to be aware of. In general you want to only copy that material inside a "transfer event". Sam Mestman discusses this from 06:30 to 11:00 in the below video. He demonstrates this on a Lumaforge NAS but it's not unique to a NAS. "Final Cut Pro X Virtual User Group #7":


There is also an FCPX media management issue if duplicate filenames are imported, even if those were within separate disk folders. FCPX places the media or symlinks within a simplified library folder structure, which creates filename conflicts. It solves this by adding a "uniqueifier" of (fcp1), (fcp2), etc. However this doesn't work correctly in all cases and if exporting/importing XMLs can create spurious duplicate clips. To avoid this (as well as good data hygiene) it's best if all media files are globally unique across all libraries and all time. To achieve this, before import you could add an incrementing 6-digital serial number, then save the high value and use the next value for the next import. Files can easily be batch renamed to add that serial number using either Finder or a 3rd party tool like A Better Finder Rename.

Finder batch renaming (MacMost):


There is also an FCPX media management issue if duplicate filenames are imported, even if those were within separate disk folders. FCPX places the media or symlinks within a simplified library folder structure, which creates filename conflicts. It solves this by adding a "uniqueifier" of (fcp1), (fcp2), etc. However this doesn't work correctly in all cases and if exporting/importing XMLs can create spurious duplicate clips. To avoid this (as well as good data hygiene) it's best if all media files are globally unique across all libraries and all time. To achieve this, before import you could add an incrementing 6-digital serial number, then save the high value and use the next value for the next import. Files can easily be batch renamed to add that serial number using either Finder or a 3rd party tool like A Better Finder Rename.

Finder batch renaming (MacMost):

A Better Finder Rename: www.publicspace.net/ABetterFinderRename/index.html

Please Log in or Create an account to join the conversation.

media asset management system 02 Aug 2020 22:41 #109087

  • DaveMaine
  • DaveMaine's Avatar
  • Offline
  • Fresh Boarder
  • Fresh Boarder
  • Posts: 2
  • Thank you received: 0
I am using FCPX as a media management tool as well. What I do differently is leave the media in place wherever it exists, and just bring in the file reference. It keeps the library much smaller in the single library.

My original footage is stored on a series of discs, in folders by year, month and day. They never move, and are routinely backed up. To use anything in a new library and project, I just bring it in as a reference movie as well.

There may well be soft limits, but I have not yet hit them. I have all the footage from 1960 to now, from 8mm transfers to UHD footage
Attachments:

Please Log in or Create an account to join the conversation.

media asset management system 03 Aug 2020 10:45 #109090

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1551
  • Karma: 27
  • Thank you received: 331

DaveMaine wrote: I am using FCPX as a media management tool as well. What I do differently is leave the media in place wherever it exists, and just bring in the file reference. It keeps the library much smaller in the single library...


You are proving it can be done. Thanks for posting that.

Even though your media is carefully placed on a certain disk, you could also store with the library a list of all media paths used. Reason: if in the future you get red "missing media" clips the first thing you want to know is where is FCPX looking for the now-missing media. Unfortunately there is no tool within FCPX or Finder which lists the path of each media file, or which displays the symlink pointers. Finder is especially frustrating because "Get Info" only resolves the symlink IF the target file is there, in which case you don't need it. When the target file is missing Finder only shows the path of the symlink itself which is useless.

In that situation you can use terminal, navigate inside the library to the original media folder and do ls -l to make it list the full pathnames the symlink is pointing to, or you can just use Final Cut Library Manager, which has an optional feature to export all this to CSV. The future person might not have this knowledge so including the CSV in the same folder as the library might be a good idea: www.arcticwhiteness.com/finalcutlibrarymanager/

Please Log in or Create an account to join the conversation.

media asset management system 03 Aug 2020 11:17 #109092

  • Tom Wolsky
  • Tom Wolsky's Avatar
  • Offline
  • Moderator
  • Moderator
  • Posts: 4490
  • Karma: 112
  • Thank you received: 733
If you try to relink a file the relink dialog I think still shows the expected filepath.

Please Log in or Create an account to join the conversation.

media asset management system 03 Aug 2020 12:59 #109093

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1551
  • Karma: 27
  • Thank you received: 331

Tom Wolsky wrote: If you try to relink a file the relink dialog I think still shows the expected filepath.

Thanks for that correction. Yes if you select a bunch of red "missing media" files, then do File>Relink Files, and pause at that screen, you can cursor down and one by one, it will show the expected file pathname at the bottom. However there's no way to obtain a comprehensive list.

But in some cases all you may need is a few file pathnames to help determine why they are missing.

This is a good recommendation because at first glance the UI is not obviously showing the expected file pathname. It is easy to miss.

Please Log in or Create an account to join the conversation.

media asset management system 04 Aug 2020 17:05 #109116

  • Glitchdog
  • Glitchdog's Avatar Topic Author
  • Offline
  • Fresh Boarder
  • Fresh Boarder
  • Posts: 3
  • Thank you received: 0
Thanks to all that have posted. It's really great to hear what others are doing and glean additional wisdom on this topic. We have been looking at Axle.ai, as it's in our price range, and a very powerful and simple MAM, but it all started when we were still focused on Premiere. Axle is developing some integration with FCPX, and currently have the ability to export out an xml of keywords that can then be imported into FCPX. So, very rough at this stage. And no specific timeline, as they are just getting enough FCPX user interest to do more development. Early stages.

I have read some about Kyno, but they say on their website: Is Kyno a Media Asset Management System? Well, sort of. Many of the features like tagging, descriptive metadata support and filtering give it a MAM kind of feel but its scope is currently rather a very light-weight support of production processes rather than long-term archival, although people may still find it useful for that as well. The main point is, that it is much more light-weight than typical MAMs because it does not require an import/ingest step before you can do something useful with your material. That means there is not really a concept of "inside" or "outside" of Kyno, which also means there is no global search of all content Kyno has ever seen. I will be contact Kyno today to get a bit more clarity. I thought you could do global searches in Kyno for all your content, etc.

It's confusing to sort metadata support all out.

The general impression I'm getting is that a number of people are using FCPX to track and manage their footage with metadata. I have a local production friend that suggested that as well. Ultimately I've been asked by my org to have something long-term in place that can eventually be migrated to the cloud. At least as much is anything is long-term.

Please Log in or Create an account to join the conversation.

  • Page:
  • 1