Welcome, Guest
Username: Password: Remember me

TOPIC: Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 01 Jul 2017 03:11 #89146

  • Smesh
  • Smesh's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 65
  • Karma: 0
Hi everyone. I searched and searched but can';t find the answers. I'm pulling my hair out. I'm producing a reality adventure travel show for television and doing the editing myself. I've been editing on a small scale off and on since FCP 7.

Most of the project was shot in 1080p and I was happily editing on a 12TB G-Speed Studio R TB2. I transcoded everything to optimized media which took forever. Since there was over 24 days of filming the 1TB of 1080p footage turned into like 6TB. Then we realized we needed to do a bunch of interviews to fill in some story gaps, and wanted to cut a bio doc for one of the celebrities. For this, we decided to shoot it all in 4k. there was like 9 days of shooting on location. Copying dailies, then transcoding to optimized in my hotel room at night. I quickly ran out of room, and started just storing the original files on standard hard drives to deal with everything when I got back to LA.

I realized I needed to either add another G-Speed, or get a much bigger array. Since the 4K was killing my MBP, I bought a G-Speed Shuttle XL 24TB. This clocked at 1100 MB/s so is much faster. I used the FCP copy library command to migrate everything to the new raid. It worked great, or so I thought. I set up the media folders the same as the other array and started to transcode all the 4k footage that wasn't done yet, but this time I started with proxy first to speed up workflow. In fact, I went through and set my library to transcode all the 1080p stuff to proxy as well. As you can guess, this took days....

Then I found problems. What I didn't do, that I should have done, was disconnect the original Studio-R. Some of the proxy files for the new library on the new larger raid, seemed to have been generated inside the old library. Not all, but some. At least how that's how I remember it. Also, many of the proxy files were inside the media folder, but outside of the proxy folder. They had no file extensions either, but if I added the .mov file extension, I could play the file. I can't recall exactly how I did it, but eventually, I deleted the proxy files that did not have extensions.

I consolidated all the media on the new raid, and it came back several times saying "nothing to consolidate" but I still had missing file icons. I disconnected the old raid, I tried relinking, I tried re=transcoding. Some files re-linked, others will not. I hit transcode the files, check one or both empty boxes and nothing happens in the background tasks. All the media seems to be in the appropriate folders now though. I just can't get it all to relink. It's mostly re-linked but still lots missing. And in many cases, the optimized media for the 1080p footage is not showing the green dot in the inspector window either. It's in the folder, but won't show up.

I'm extremely worried about the status of my library and work so far. The old library seems okay, but I can't continue to work on that because I'm out of space, I also have done a ton of work already, and I am really hoping I won't have to reimport and transcode everything all over again. There are hundreds of hours of footage. One thing I read about, is the whole drive name scenario. I did not use the same name when I copied the library and files over, and I'm afraid to do that now. Any help would be greatly appreciated.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 01 Jul 2017 05:44 #89150

  • Smesh
  • Smesh's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 65
  • Karma: 0
Okay. Some of my optimized files are missing inside the folders, and I cannot force FCPX to transcode new files from the originals either. Any ideas? Really could use some help here.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 01 Jul 2017 12:20 #89159

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2778
  • Thank you received: 367
  • Karma: 34
First, if you optimize there is no need to generate proxies. Why did you optimize in the first place? What cameras were your clips coming from?

What you should have done before disconnecting the other drive was a Consolidate command. Verify the storage settings for the new Library are all set to store everyone inside the Library itself, then consolidate.

Did you have "Copy To Library" or "Leave In Place" set when you imported footage?

If you had "Copy To Library" set, then generated Optimized files, then generated proxies, that's so much needless excess, and yes, will eat up untold terabytes unnecessarily.

If you have the old drive, plug it in, launch the new library copy, and consolidate.

You simply could have done a drag-and-drop to copy the library in the Finder, too. Much easier.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 01 Jul 2017 15:14 #89164

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1244
  • Thank you received: 258
  • Karma: 22
Smesh wrote:
....Any ideas? Really could use some help here.

This is a complex situation. You can contact me directly at joema4 at gmail dot com and I'll try to help you.

Until then I suggest not doing anything further on your new RAID and don't do anything on the old RAID to jeopardize your data or library.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 01 Jul 2017 20:34 #89167

  • Smesh
  • Smesh's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 65
  • Karma: 0
FCPX.guru wrote:
First, if you optimize there is no need to generate proxies. Why did you optimize in the first place? What cameras were your clips coming from?

I optimized the 1080p footage because I wanted better performance. I'm on a maxed out late 2013 MBP with 16GB RAM. Most of the files were .mp4 files from a Sony NX100 shot in 1080p 59.97p, but some were from another camera. Not sure the model, that we .mxp files and they did not play well when compressed. There was also a fair bit of gopro footage. We're talking over 1000 clips. After optimizing them, they played fine on my computer. It was after we shot another almost three weeks in 4k that I had issues. Both with storage and playback. A few days with a FS5 and several days with GH5's in 4K 60p, 4k 24p 10 bit, and 1080p 180fps. I optimized some, but the transcoding was taking days and days and editing bogged down in fcpx. I decided to try to work in proxy, so having the room on the new raid, tried to transcode everything to proxy. Because the 4k footage is intercut with the 1080p footage, I could not work in proxy mode without having proxy files for the 1080p footage.
FCPX.guru wrote:
What you should have done before disconnecting the other drive was a Consolidate command. Verify the storage settings for the new Library are all set to store everyone inside the Library itself, then consolidate.

Did you have "Copy To Library" or "Leave In Place" set when you imported footage?

If you had "Copy To Library" set, then generated Optimized files, then generated proxies, that's so much needless excess, and yes, will eat up untold terabytes unnecessarily.

I consolidated, then copied to library and included everything. I had 24TB of storage, so wanted to bring over everything and keep the backup on the old raid drive (which I might sell once I feel secure that I have everything working ands backed up or even after the project is delivered). Then I consolidated the new library again. Only then, did I start importing and transcoding more 4k footage. I had run out of room on the old raid and backup drives that I took to our shoot location out of state, so had to buy more. I knew I was going to buy a bigger raid, so just stopped transcoding until I got home, bought the new array, and had everything set up.
FCPX.guru wrote:
If you have the old drive, plug it in, launch the new library copy, and consolidate.

You simply could have done a drag-and-drop to copy the library in the Finder, too. Much easier.

I have dragged and dropped before, but understood that the copy to library was the recommended and safest option.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 01 Jul 2017 20:34 #89168

  • Smesh
  • Smesh's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 65
  • Karma: 0
joema wrote:
Smesh wrote:
....Any ideas? Really could use some help here.

This is a complex situation. You can contact me directly at joema4 at gmail dot com and I'll try to help you.

Until then I suggest not doing anything further on your new RAID and don't do anything on the old RAID to jeopardize your data or library.

Thanks Joe. emailing now.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 01 Jul 2017 21:59 #89169

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2778
  • Thank you received: 367
  • Karma: 34
I've done reality TV on a 2015 MBP with a RAID, and we imported native codec, using proxies when needed. No need for optimized at all, only proxies, takes up less room, cause less headaches.

Anyway, any go to your Library, right-click, "Show Package Contents". Then you can check to see if the media is really there, or is only a link (movie file icon with hooked arrow symbol on lower left), or is simply missing. That's the first step to take. What you find will determine what to do next. Not overly complicated IMHO.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 01 Jul 2017 23:04 #89170

  • Smesh
  • Smesh's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 65
  • Karma: 0
FCPX.guru wrote:
I've done reality TV on a 2015 MBP with a RAID, and we imported native codec, using proxies when needed. No need for optimized at all, only proxies, takes up less room, cause less headaches.

Anyway, any go to your Library, right-click, "Show Package Contents". Then you can check to see if the media is really there, or is only a link (movie file icon with hooked arrow symbol on lower left), or is simply missing. That's the first step to take. What you find will determine what to do next. Not overly complicated IMHO.

My media is outside the library, but in the same folder as the library. All the media is there.

Thanks to Joe for walking me through some ideas on the phone. Just pulling a backup of the library now, then trying to few options. Agreed on the working in proxy. If I had done that from the beginning, I probably wouldn't have needed to buy a 24TB Shuttle XL. Just to give you guys an idea, I have almost 3 TB of original media.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 02 Jul 2017 03:41 #89172

  • Smesh
  • Smesh's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 65
  • Karma: 0
Just a quick note to say thank you all. Everything is fixed!!

Special shout out to Joema, who spent probably 30 minutes or more on the phone with me figuring things out, coming up with a strategy and giving me his wisdom and experience and the confidence I needed to tackle this problem.

I had a backup ready to go, but it really just came down to renaming the the new Raid array to be identical to the original, then once I did that, FCPX suddenly let me start transcoding again, and it obviously just simply reconnected much of the existing proxies files and optimized files since it went pretty fast. Fast as in took me most of the day but still way faster than if I was starting over from scratch.

Thanks again everyone! I'm back on track to get this monster project done.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 02 Jul 2017 03:42 #89173

  • FCPX.guru
  • FCPX.guru's Avatar
  • OFFLINE
  • Platinum Boarder
  • bbalser.com
  • Posts: 2778
  • Thank you received: 367
  • Karma: 34
If media is outside the Library, seems a simple re-link would solve the issue.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 02 Jul 2017 04:03 #89174

  • joema
  • joema's Avatar
  • OFFLINE
  • Platinum Boarder
  • Posts: 1244
  • Thank you received: 258
  • Karma: 22
FCPX.guru wrote:
If media is outside the Library, seems a simple re-link would solve the issue.

Relink doesn't work relliably if using external proxies and if the volume name has changed.
The administrator has disabled public write access.

Moved Large Library to Larger Raid, Now Cannot Relink or Re-Transcode But Files Are All There 02 Jul 2017 04:14 #89175

  • Smesh
  • Smesh's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 65
  • Karma: 0
Relink would not work, and trying to transcode again wouldn't work. Renaming the drive was the trick, although a few random clips still seem to have red triangles under "optimized" even though they are in the folder. It won't effect anything though as the proxy is there and it'll sort out when I export. I did read about the drive name issue, but was scared to touch anything until I talked to Joe. Plus, he had some backup plans that would have surely fixed the problem one way or the other. He is very humble but definitely knows what he's talking about. and was extremely generous with his time. Thanks again Joe.
The administrator has disabled public write access.