Importing FX9 (Leave Files In Place) With or WITHOUT XML?

Just wrapped on a corporate project that was shot with a FX9 and A7S III. The files were dumped from the cards and organized to preserve the card meta data; however, I was not able to import using "leave files in place" unless I segregated the .MXF files. So....I went through the process of stripping the XML files into their own folders and imported them without their XML or card data. 

Of course now I cannot see any of the file information including the log profile that was used for each clip. This makes it harder to choose base rec709 conversion LUTs, which I would like to do to save time. There may also be other issues I haven't realized yet. I have tried the SLOG 2 and 3 LUTS using trial and error and my windows are getting blown out.  They were not so on the SmallHD monitor using the rec709 monitoring LUT the DP was using. 

What else have I lost by doing it this way?  Should I start with a fresh library before I get too deep?  So far, all I've done is imported four interviews. I would really prefer NOT to have to import all the files into one giant library file, but if that's what it takes I will.  Seems stupid that I cannot leave files in place when importing from a "card" that's actually sitting on my raid.  Any advice would be appreciated. 

Smesh wrote: FX9...not able to import using "leave files in place" unless I segregated the .MXF files. ...stripping the XML files into their own folders and imported them without their XML or card data...Should I start with a fresh library before I get too deep?  So far, all I've done is imported four interviews. I would really prefer NOT to have to import all the files into one giant library file...

Rewrap the files with EditReady2 before importing with "leave files in place". This is very fast and should preserve all metadata. It also can span clips and allows inspection and editing of metadata:  EditReady2

Hi Joe.

I notice that EditReady hasn't been updated in 6 years. Is it still relevant today? I did some more testing and updated fcpx and was able to leave files in place. I also planned to edit without converting to prores or proxy, but it's been a while since I edited Sony footage on my base model iMac Pro, and wonder if I should just convert. I will need to send this over to a motion graphics and VFX editor, so that's also a concern. Might even try a round trip to resolve for color grading, but last time I did that, XLM round trip kept failing. Hopefully those issues are fixed, but I wonder if Editready is still work buying today?

Smesh wrote: I notice that EditReady hasn't been updated in 6 years.

It is actually updated fairly frequently. Below are some examples. I think there were more updates than these but they don't always mention it on their blog:  www.divergentmedia.com/blog/

This forum post has most of the updates, including change logs:  www.videohelp.com/software/ClipWrap/version-history

2.6.5 - August 5, 2020
2.6.4 - December 10, 2019
2.6.3 - October 7, 2019
2.6 - August 5, 2019

Smesh wrote: ..Is it still relevant today? I did some more testing and updated fcpx and was able to leave files in place....

It's relevant if you want to properly import tree-oriented media "in place" without possible downstream repercussions. In some cases you can get away with it. I've imported hundreds of terabytes of Sony 4k XAVC-S "in place" without problems. Other codecs can cause minor issues. Some such as AVCHD can cause major performance problems. 

My doc team has five A7SIIIs (same camera you're using), and we are still doing workflow testing. For the past two years, we've used only ProRes 422 acquisition via the Ninja V, BMD Video Assist and Shogun 7, but we still use the internal codec for backup. The A7SIII has several new codecs which have unique characteristics and must be individually tested. When we get the FX6 cameras we'll also have to repeat some of that.

Smesh wrote: ...I also planned to edit without converting to prores or proxy, but it's been a while since I edited Sony footage on my base model iMac Pro, and wonder if I should just convert...

On my 10-core Vega64 iMac Pro, most 4k Long GOP codecs are too sluggish without proxies, esp. for multicam. For scripted work maybe it's OK, but for documentary work with a high shooting ratio and many multicam interviews, it's just too slow. Each codec has different performance characteristics on each *version* of each NLE and hardware platform.

Smesh wrote: ...I will need to send this over to a motion graphics and VFX editor, so that's also a concern. Might even try a round trip to resolve for color grading, but last time I did that, XLM round trip kept failing...

Whether you use FCP proxies should not affect turnover to VFX or color. The proxies are a performance tool for editing. Typically VFX/color would need full-res original source files. 

I have limited experience with turnovers between FCP and Resolve, but the XML works fairly well if using a well-defined workflow. But many things will not transfer, including color wheels, color curves, masks, effects, generators, transitions (beyond basic cross fade), etc. If you and the downstream people plan a well-orchestrated workflow, stick to what works and avoid what does not, then it can work.
www.partnersinpost.com/blog/2019/2/1/rou...ci-resolve-15-part-1

When I import footage from my Sony F5 or the FX9, I use Hedge to offload the cards to a Raid (or you can Opt-drag the card to your storage location to make an exact copy of your card and its folder structure). Once offloaded, I have set my import preferences in FCPX to "Leave in Place" and I drag all the offloaded cards that I need for the project and drop them on the Event in the Browser. I also have Keywords from Folder Names checked. When I have dropped them on the Event, a dialog window pops open warning you that FCPX cannot import some files and it shows them in the dialog window. These are the .XML and .BIM files. You can click on "Continue Import".
Within seconds all your MXF files are available in your project without rewrapping. The only task FCPX has to finish in the background is the usual generation of thumbnails and waveforms.

Hope this helps.

I didn't realize I need to option click to copy the cards.  Never even thought of hidden files being an issue. I just copied the entire card contents to my SSDs and then copied those to my raid when I got back to the office. I wonder what I lost...

Never heard of Hedge either, but I will look into it.  I do have carbon copy cloner. Perhaps I should be using that instead.

For some reason, FCPX 10.5.1 is not creating smart collections based on folder names. I did a test and duplicated all. the footage into a new folder, then. organized it all into sub folders with keyword rich folder titles and did the same with the file names leaving the original camera file appended on the end. FCPX is not creating keywords at all. I will have to do them all manually.

Make sure you have checked the tickboxes to create Keywords from Folders and from Finder Tags in the FCPX Preferences under the Import Tab.
Hedge is a very good program for offloading your cards. It makes a verified copy of your card and leaves the folder structure in tact. You can have the program give a name to your card with an automatically generated date and a counter. You can make presets for different productions etc. Just pop your card in the reader and it will offload, even to multiple destination disks if you want to have a safety backup. It will show a green bar when the card has been succesfully copied and verified. And it is fast! A 128GB card copies in 5 to 6 minutes to two Samsung T5 SSD's simultaneously.

joema wrote:

Smesh wrote: FX9...not able to import using "leave files in place" unless I segregated the .MXF files. ...stripping the XML files into their own folders and imported them without their XML or card data...Should I start with a fresh library before I get too deep? So far, all I've done is imported four interviews. I would really prefer NOT to have to import all the files into one giant library file...

Rewrap the files with EditReady2 before importing with "leave files in place". This is very fast and should preserve all metadata. It also can span clips and allows inspection and editing of metadata:  EditReady2

joema I was having the same issue as the OP and use EditReady2. However, while this solution seems to solve the issue, does this mean that you then delete the original camera cards to save space? It doesn't completely solve the massive doubling of data issue where you backup your card to disk and then need to double up that data to use within FCPX. So I'm curious if you feel safe once you've rewrapped in EditReady2 to dispose of these originals without the file structure from the SD or XQD cards? This isn't traditionally how I've worked, but I know things are constantly changing.

That is a good question. My team avoids this by only using ProRes 422 acquisition -- we have Atomos recorders on all Sony Alpha mirrorless cameras, inc'l the FS5 and FX6.

In general I'd feel OK about only keeping re-wrapped files from EditReady2. The re-wrapped files in some cases are easier to browse at the Finder level,  but some of the new A7SIII 10-bit 4:2:2 codecs still won't play in Quicktime or Quick Look, despite being re-wrapped. So it's an advantage to rapidly import those to FCP with "leave files in place" for browsing.

However, a new problem is the Catalyst gyro data is only present in the original camera files. If we shot to the internal codec with the intention of Catalyst stabilization in post, I don't think that would work with re-wrapped files from EditReady2. We haven't yet decided how to handle that.

Ah that's interesting. That would apply to both the FX9 and all newer FX and Alpha cams for handheld footage. I thought the whole point of FCPX getting the 10.4.10 update was to add FX9 support but it seems like we're back to the old method like FCP7 of transcoding/rewrapping and having twice the media instead of the promised simplicity of editing straight from camera originals.

truelife2 wrote: I thought the whole point of FCPX getting the 10.4.10 update was to add FX9 support but it seems like we're back to the old method like FCP7 of transcoding/rewrapping and having twice the media instead of the promised simplicity of editing straight from camera originals.

I think the FX9 support meant FCP would import and understand the files, vs a black or red thumbnail. It did not mean any change to the prior FCP data handling approach of copying to library and re-wrapping all tree-format media. I don't know the technical reason for that because Resolve and Premiere don't copy and re-wrap tree-format media during import.

Ah ok understood. So basically, the rule should be keep only files which have essential metadata for Catalyst etc, and dispose of the rest, using rewrapped files from EditReady as Masters. 

The other issue that continues to cause media manage headaches is that I have a ton of multicam clips and each of those requires that the media be optimized. I'm now wondering if it just makes more sense to transcode to ProRes from the outset to avoid having to deal with it happening during multicam creation in FCPX?

There is no simple answer. My team will likely keep using ProRes 422 and shoot simultaneously to the internal codec for backup, and only keep the internal files which are intended for Catalyst stabilization.

Yes multicam is slow on many 4k Long GOP codecs, esp. the new 10-bit 4:2:2 ones. That's not just FCP, all NLEs are slow. I just tested various internal A7SIII internal codecs and FCP (while sluggish) was significantly faster than Resolve Studio 17.1, at least on my iMac Pro.

Apple Silicon is different and there's hope the next generation of machines will have improved decoding performance on 10-bit 4:2:2 Long GOP codecs and maybe enable 4k multicam editing on the original camera files.

Another option starting with FCP 10.4.9 is have the cameras generate proxies, import and relink to those. That avoids any time spent transcoding proxies. I tested this on A7SIII files and it works well but I had to rename the proxy files in a separate folder to have identical filenames as the 4k files. FCP also has "proxy preferred" which is a big improvement over the previous "all or nothing" proxy mode.

Yes, but again with proxies- that's more data on the front end as well which would use up card space quicker. Of course if you are running Atomos in the field then it's probably less of an issue, but I'm just a bit frustrated that I can't just shoot and import and edit similar to the other NLEs, but your point about Resolve is well taken. I too am on a iMac Pro with a TB3 Raid 5, but I find it just gets bogged down easily, particularly when cutting 4k multicam clips. I'd be interested to use an SSD Raid to see if the performance would improve drastically.

truelife2 wrote: Yes, but again with proxies- that's more data on the front end as well which would use up card space quicker...I too am on a iMac Pro with a TB3 Raid 5, but I find it just gets bogged down easily, particularly when cutting 4k multicam clips. I'd be interested to use an SSD Raid to see if the performance would improve drastically.

The FX9 and FX6 proxy codec is 1080p 8-bit 4:2:0, only about 1/10th the size of the 4k XAVC-I codec, so space really isn't that bad.

The size difference is less between proxy vs 4k XAVC-L, which is only 8-bit 4:2:0, but I'm not sure how many FX9/FX6 operators will shoot that codec. The A7SIII will do 4k 10-bit 4:2:2 Long GOP but the FX9 and FX6 will not.

Re your iMac Pro getting bogged down, IF original camera files that is more likely a decode performance issue than an I/O issue. OTOH if you have already transcoded to 4k optimized media, the IO load can add up. But it's normally only about 50 megabytes/sec per camera angle, assuming 4k ProRes 422 at 24 fps. Most 4-drive mechanical arrays should be able to handle 2-3 angles. I have several 4-drive RAID-0 mechanical arrays and two 4-drive SSD RAID-0 arrays. The mechanical arrays aesthetically sound like they're slowing down editing because of the noise of head seeking. But when moving that data to my SSD array it's quieter but oftentimes not dramatically faster.

You ideally want the library and cache on an SSD to reduce interference with the sequential I/O streams from the mechanical RAID.

Well I also sold, produced and directed this project. I hired the DP, who use the FX9 and A7Siii. The guy was fantastic, but having dealt with the footage, I will not hire another DP with Sony gear. I have worked with a Sony project before, and it had issues as well, but that was shot late 2018 on FS7 (ii I think), the A7Sii and A7R iii. I hoped the FX9 and A7SIii along with Final Cut Pro would not have any issues.

I also later realized that the log from the FX9 is not wide gamut. That sucks as I prefer to have everything delivered in HDR these days.

Anyway, I will NOT use Sony products in future productions. 

This is really helpful- thank you. Out of curiosity, why do you choose RAID 1 or 0 over 5? I always understood 5 to be the standard just for the redundancy?

And just to confirm- you are suggesting keeping a separate SSD drive just for the library and keeping footage on a different drive from the library?

truelife2 wrote: why do you choose RAID 1 or 0 over 5? I always understood 5 to be the standard just for the redundancy?

While RAID-5 is protected from a single physical drive failure, there are various other risks of total data loss. E.g, filesystem bug, malware, chassis failure, application-layer database corruption, etc.

While there is significant variation, some RAID-5 systems enter a severely degraded performance mode during the rebuild phase, which in some cases can take days. In general the content must be backed up anyway, and you may need rapid recovery to usable performance. E.g, if RAID-5 had a drive failure, you replace the failing drive, then it starts rebuilding but it's too slow to use for 48 hr, that may not be practical from a business standpoint.

With dual RAID-0 arrays, I keep one off line and sync them about once per day. That way if one fails I can recover within minutes to a full-performance state. The FCP library itself holds the edits and I keep that backed up several places.

truelife2 wrote: ...you are suggesting keeping a separate SSD drive just for the library and keeping footage on a different drive from the library?

Yes ideally a "lean library" (containing no media or cache) should be kept separately on SSD. That enables easy Finder-level duplication for backup, plus separates the media IO stream (which is largely large sequential reads) from the database IO stream which includes lots of small random IOs. The cache can also be kept on the SSD, since lots of small random IOs are done there for thumbnails, waveforms, plist files, plus render cache. It could be kept inside the library on SSD, but it's easier to manage if in a separate external folder. E.g, there's no way in the FCP UI to delete thumbnails or optical flow analysis files. If kept in a separate cache folder that can be safely deleted using Finder.

joema wrote:

truelife2 wrote: why do you choose RAID 1 or 0 over 5? I always understood 5 to be the standard just for the redundancy?

While RAID-5 is protected from a single physical drive failure, there are various other risks of total data loss. E.g, filesystem bug, malware, chassis failure, application-layer database corruption, etc.

While there is significant variation, some RAID-5 systems enter a severely degraded performance mode during the rebuild phase, which in some cases can take days. In general the content must be backed up anyway, and you may need rapid recovery to usable performance. E.g, if RAID-5 had a drive failure, you replace the failing drive, then it starts rebuilding but it's too slow to use for 48 hr, that may not be practical from a business standpoint.

With dual RAID-0 arrays, I keep one off line and sync them about once per day. That way if one fails I can recover within minutes to a full-performance state. The FCP library itself holds the edits and I keep that backed up several places.

truelife2 wrote: ...you are suggesting keeping a separate SSD drive just for the library and keeping footage on a different drive from the library?

Yes ideally a "lean library" (containing no media or cache) should be kept separately on SSD. That enables easy Finder-level duplication for backup, plus separates the media IO stream (which is largely large sequential reads) from the database IO stream which includes lots of small random IOs. The cache can also be kept on the SSD, since lots of small random IOs are done there for thumbnails, waveforms, plist files, plus render cache. It could be kept inside the library on SSD, but it's easier to manage if in a separate external folder. E.g, there's no way in the FCP UI to delete thumbnails or optical flow analysis files. If kept in a separate cache folder that can be safely deleted using Finder.

I just wanted to thank you for these tips again. After reading this, I purchased an OWC 4TB SSD Raid 0 and my projects are flying like never before. Really appreciate it!

I have two RAIDS, both level 5, super great performance, had them for years, no issues. The " filesystem bug, malware, chassis failure, application-layer database corruption, etc." stuff may be technically accurate, but in 20 years of media production, and the 20 years of IT before that, never saw any of that happen to a RAID.