fbpx
Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1

TOPIC: Needed: How to Setup External Hard Drives For FCPX

Needed: How to Setup External Hard Drives For FCPX 28 Feb 2015 17:54 #60603

  • RayPaula
  • RayPaula's Avatar Topic Author
  • Offline
  • Expert Boarder
  • Expert Boarder
  • Posts: 99
  • Thank you received: 2
Hi, I'm about to purchase a iMac 5k for video editing configured as;

Hardware:
4.0GHz Quad-core Intel Core i7, Turbo Boost up to 4.4GHz
32GB 1600MHz DDR3 SDRAM - 4x8GB
1TB Flash Drive (SSD)
AMD Radeon R9 M295X 4GB GDDR5

Storage:
PROMISE Pegasus2 R4 8TB (4 by 2TB) Thunderbolt 2 RAID System
store.apple.com/us/product/HE...r4-8tb-4-by-2t ...

Software:
Final Cut Pro X
Motion 5
Compressor

I need some instruction on how to setup/configure RAID 0 or RAID 5 for video editing with FCPX. I'm coming from a PC with Windows 7, Vegas Pro. I'm using two internal hard drives (2TB each) in RAID 0. These drives are used as follows;

Internal Hard Drive 1: Raw un-edited pre-rendered video, pictures and music (2x750GB Raid 0) Samsung HD753LJ 750GB SATA2 7200rpm 32MB

Internal Hard Drive 2: Rendered events for final output. (2x750GB Raid 0) Samsung HD753LJ 750GB SATA2 7200rpm 32MB

I do my editing on the computers OS drive. I really don't know if this setup is proper. I want an optimal setup for use with FCPX, Motion 5 and Compressor. Are there any instructional tutorials and/or videos out there that will work me through a setup process from start to finish? If so, please let me know. Thanks in advance. Ray

Please Log in or Create an account to join the conversation.

Needed: How to Setup External Hard Drives For FCPX 01 Mar 2015 17:12 #60645

  • VidGreg
  • VidGreg's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 973
  • Karma: 28
  • Thank you received: 195
Hi Ray,
The Pegasus is a hardware RAID unit. You set-up RAIDs via the software provided by Promise Tech.
Slight confusion in using the term "Internal Hard Drive". Internal generally refers to internal to the computer. In your case you are referring to external RAIDs 1&2 as configured in the Pegasus. Not to be confused with a RAID 1 array.
RAID5 requires 3 or more drives.
Not sure that you will see any speed improvements by separating rendered vs non-rendered media files. I suspect the export will be the same speed just using a RAID 0 array. You could speed up everything by increasing the number of drives in the array. I haven't tested this so someone else should chime in.
The other thing to remember is that RAID even 1 or 5 or any other configuration should not be considered as a valid back-up. You need separate drives for that. This is mainly to prevent user error along with hardware failure. Hardware based RAIDS also require an identical device to restore to if the unit itself dies. RAID0 provides no restore ability if (when) a drive itself dies. Having rotating back-ups to 2 or more drives or cloud based back-ups is a good idea. Many people keep at a least 1 drive at different location in case of disaster.
File and library management along with workflow plans will most likely result in happy & speedy editing. MacBreak Studios and others have video tutorials on YouTube and Vimeo.
Congratulations on your new system.
Hope this helps, Greg

Please Log in or Create an account to join the conversation.

Needed: How to Setup External Hard Drives For FCPX 01 Mar 2015 17:58 #60646

  • RayPaula
  • RayPaula's Avatar Topic Author
  • Offline
  • Expert Boarder
  • Expert Boarder
  • Posts: 99
  • Thank you received: 2
Hi, Thanks for the help.....

Slight confusion in using the term "Internal Hard Drive". Internal generally refers to internal to the computer. In your case you are referring to external RAIDs 1&2 as configured in the Pegasus. Not to be confused with a RAID 1 array.

The internal hard drives are in relation to my Windows PC workstation which are setup in RAID 0. Sorry for the confusion.
I realize that RAID 0 is not a safe RAID to be in simply due to the possibility of a crash. It appears that RAID 5 may be the way to go. My biggest problem is that I'm oblivious in understanding why it’s important to have good fast external hard drives, for example as the Pegasus. Please correct me if I’m wrong, I believe that all the editing and rendering has to be performed on the OS drive? I know I’m missing something in the mix and I sure could use some help. I will take a look at MacBreak Studio's in hoping to find some additional answers. Finding out what each external hard drive is used/assigned for and their purpose would clear my mind. Thanks again for all your help. Ray

Please Log in or Create an account to join the conversation.

Needed: How to Setup External Hard Drives For FCPX 01 Mar 2015 21:52 #60650

  • VidGreg
  • VidGreg's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 973
  • Karma: 28
  • Thank you received: 195
Hi Ray,
The reason it is recommended to use external drives for your media is to free up some bandwidth and resources. The external drives will feed the media streams mostly into system ram or vram on the GPU, where the apps on the boot drive will process the edits. The boot drive also has to deal with all apps open, system maintenance etc… Lots of data being moved around. Off loading the data stream to the external drive frees up system resources for more efficiency.
Also, video eats up lots of space, so having external drives makes it easier/cheaper to store hours of video.

ALL RAIDS should not be considered as a back-up strategy, no matter what set-up. RAID 5 provides some safety in that it creates what is known as parity (not the same as a mirror), meaning that it can rebuild the data if a drive fails, but this is not the same as a Back-Up. This is something lots of people do not understand. Back-ups are periodic snap shots of your data. RAID is always dynamic and subject to immediate change. If you delete something, it is gone!

Setting all drives to RAID 5 may be a way to go for you, if you need the throughput speeds. How much data throughput you need is largely based on type of video you shoot (HD/2K/4K) how many cams, effects, etc…
Here is a pretty good explanation by Larry Jordan… www.larryjordan.biz/app_bin/wordpress/archives/2559

Hardware RAID like in the Pegasus is dependent on the actual device (enclosure); the chipset on board, the firmware, the set-up software. If the unit itself dies; say the controller chip dies, then you need an empty EXACT duplicate of the unit including the same chipset, firmware, software in order to retrieve the data on the drives(hdds/ssds) themselves.
If a drive dies while using RAID 1, 5, 10 and several others, then you can replace the drive itself and restore and retrieve the data so long as the unit itself is functioning.

RAID 0, lose a drive, data is gone! PERIOD! Only thing to do is erase good drive. Same goes for drive data corruption.

So use a back-up plan. Make copies of all your media, edits, etc… so you can recover WHEN something happens.

Some videos you may want to watch include MacBreak Studios Episodes 250/251/252 all on YouTube. While these talk about a nMP not iMac, still some good info about Pegasus RAID, why external media, and other good info. You can check out some other videos they have on the PixelCorps / MacBreak Studio channel… www.youtube.com/user/pixelcorps

Hope this helps, Greg

Please Log in or Create an account to join the conversation.

Needed: How to Setup External Hard Drives For FCPX 02 Mar 2015 13:32 #60703

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1666
  • Karma: 27
  • Thank you received: 361

RayPaula wrote: ...It appears that RAID 5 may be the way to go....I believe that all the editing and rendering has to be performed on the OS drive?...


You have a good system and will be happy with it. The R4 is a 4-drive array so using RAID5 is a good idea. Unlike some RAID5 systems which are slower on writes, the R4 is very fast on writes.

However the R4 RAID initialization can take a long time. It's only needed one time on first use. This is a function of stripe size you select. See my tests and graphs here: forums.macrumors.com/showpost.php?p=18576138&postcount=1

You select the R4 configuration using the Promise utility they provide.

It's a good idea to have a spare R4-compatible RAID5 drive on hand. While RAID5 allows work to continue while rebuilding a failed single drive, you want to minimize the time in that degraded mode. Having a spare drive on hand (along with any needed caddy or mounting rails) facilitates that.

Where you place your content vs libraries is a personal choice. In general I would not suggest creating multiple partitions on the R4, just use one big partition.

For most H.264 video the actual I/O rate is fairly low (unlike raw or ProRes 422). You could experiment with putting everything on the R4 or putting your libraries on the 1TB SSD. However libraries can rapidly balloon to huge sizes if using optimized media.

Some handy inexpensive utilities: iStat Menus: bjango.com/mac/istatmenus/
Final Cut Library Manager: www.arcticwhiteness.com/finalcutlibrarymanager/
Beyond Compare file/folder comparison tool: www.scootersoftware.com/
Carbon Copy Cloner: bombich.com/

As Greg said you must have one or more backup drive, separate from the R4 RAID array. This can be large enough for a full system backup, or just your media and libraries. An 8TB G-Raid is about $750 and would back up everything very fast: www.amazon.com/G-Technology-Thunderbolt-...&keywords=8tb+g-raid

In general I suggest having at least two independent backups made different ways. E.g, a constantly-connected drive for Time Machine backups, and a separate drive you only connect periodically for Carbon Copy backups. Of course those should be separate from the R4 RAID array.

You can also do selective backups to bus-powered portable USB 3 drives. Most of them are pretty slow but the 1TB HGST Touro S is pretty fast (about 130 MB/sec): www.touropro.com/en/product/touro-s/

Likewise the 2TB Seagate Backup Plus Slim is pretty fast for that class: www.amazon.com/Seagate-Backup-Portable-E...ate+backup+plus+slim

I always do a burn-in test on any new drive using either Scannerz: scsc-online.com/Scannerz.html

Or DiskTestR: diglloydtools.com/disktester.html

Please Log in or Create an account to join the conversation.

Last edit: by joema.

Needed: How to Setup External Hard Drives For FCPX 02 Mar 2015 14:35 #60704

  • RayPaula
  • RayPaula's Avatar Topic Author
  • Offline
  • Expert Boarder
  • Expert Boarder
  • Posts: 99
  • Thank you received: 2
Hi Greg,
First, I thank you for taking the time for helping me out.

The external drives will feed the media streams mostly into system ram or vram on the GPU, where the apps on the boot drive will process the edits.

Please correct me if I'm wrong..... I think I got it....
In other words, un-edited media that's stored on an external drive is being streamed/fed on the GPU, therefore a fast external drive is essential. In return, that information is sent to the app on the boot drive (FCPX) for editing. This makes a much better process than having all your media on the boot drive only.

Setting all drives to RAID 5 may be a way to go for you, if you need the throughput speeds. How much data throughput you need is largely based on type of video you shoot (HD/2K/4K) how many cams, effects, etc…

I will be editing HD1080, XAVC L (GOP) and 4K. I "will not" be editing in Raw. My equipment is as folows;
Sony XDCAM PXW-X70 w/ 4K update on the way
Canon EOS 7D
Go Pro Hero 2
I do like using some effects, chroma keying, transitions, music. My final event edits range from 10 minutes to an hour. With that being said, how would you setup RAID for the Pegasus R4 8GB (4x2TB) or a Pegasus R6 12GB (6x2TB) drive? I believe the R6 is about a third faster than the R4. I would much rather run in a safer RAID, but I don't want to give up speed.... I guess I want my cake and eat it to....LOL. Basically I want a good workflow without hicups/lagging while working on multible timelines. I do realize, with the iMac you suffer from fast rendering due to the lack of cores found in the Mac Pro's. Rendering times are not important to me simply due to no real deadlines. I normally render overnight.

So use a back-up plan. Make copies of all your media, edits, etc… so you can recover WHEN something happens.

Due to using RAID 0, I always have saved my media to other external drives "not" associated with my workstation. I do feel making backups in any RAID configuration is the smart thing to do.
I will take a look at Larry Jordans explanation as well as the MacBreak videos. I can't thank you enough for all your help and pointing me in the right direction for a proper setup. "HUGE" THANK YOU! I sincerely appreciate it. Ray

Please Log in or Create an account to join the conversation.

Last edit: by RayPaula.

Needed: How to Setup External Hard Drives For FCPX 02 Mar 2015 15:12 #60707

  • RayPaula
  • RayPaula's Avatar Topic Author
  • Offline
  • Expert Boarder
  • Expert Boarder
  • Posts: 99
  • Thank you received: 2
Hi Joema,
Thanks for taking the time for helping me out....
As both you and Greg has stated; "RAID 5 may be the way to go" It does appear that this would give fairly fast speed along without losing all if there is a problem. With that being said...Backups are essential.
I took a look at your test on the R4, looks to be great with both write and read. I'm also considering the R6 12TB (6x2TB) Pegasus simply due for some added storage along with faster write/read times. I'm only hesitant due to everything being in one enclosure. As you stated, a spare R4 or G-RAID drive may be the way to go. It would make a nice backup along with some storage capability.
I would rather keep all my media, libraries and renders off my OS boot drive. Even though it's 1TB as you pointed out it could rapidly balloon. With 1080 and 4K content, it probably wouldn't take long. Although, experimenting could be interesting.
I will take a look at the links you provided. As with Greg, I can't thank you enough for all your help and pointing me in the right direction for a proper setup. "HUGE" THANK YOU! I sincerely appreciate it. Ray

Please Log in or Create an account to join the conversation.

Last edit: by RayPaula.

Needed: How to Setup External Hard Drives For FCPX 18 Aug 2015 04:29 #67529

  • shanesangle
  • shanesangle's Avatar
  • Offline
  • User is blocked
  • User is blocked
  • Posts: 12
  • Thank you received: 0
Hello again, Joema,

I found this thread and appreciated you detailed info - including the link to the Pegasus R4 Performance Tests.

I purchased a Pegasus R4 (4 x 4TB) DISKLESS as the price was significantly cheaper than the pre-configured R4 with 4x2TB. This decision was made after reading various threads on fcp.co including some direct comments from you - thanks!

I am wondering what STRIPE settings to use for video editing: is a Stripe size of 512kb optimal? I intend to setup the R4 in a RAID5 configuration.

I will be using FCP X 10.2 and editing AVCHD clips from a Canon C100 which I will import as managed media. The lengths of most of my clips (interviews) are between 1 and 2 hours.

Once again, your help would be greatly appreciated.
Shane
Attachments:

Please Log in or Create an account to join the conversation.

Last edit: by shanesangle.

Needed: How to Setup External Hard Drives For FCPX 18 Aug 2015 13:38 #67552

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1666
  • Karma: 27
  • Thank you received: 361
I am using a 512KB stripe size on my 8TB Pegasus R4 RAID5 array. At the time of my original testing there was a huge difference in RAID *initialization* time at different stripe sizes, but not so much difference in actual performance. In general I think 512KB is a good setting for video editing, but it probably wouldn't make much difference if you chose 256KB or 1MB.

As mentioned previously the I/O demands of 1080p H.264/AVCHD aren't that high. You can see that from the camera bit rate on the C100 or similar cameras -- a max of 35 megabit/sec (about 4.3 megabytes/sec) on the C100 Mark II using MP4. That's for a single stream; for multicam it will be higher, but not that high. If you transcode to ProRes the I/O rate will be much higher, but it lowers the CPU load and sometimes can make rapid JKL editing a bit more responsive. However in general you can edit the unoptimized camera files. AVCHD files cannot be edited in place in FCP X and they will be copied on import but it's up to you whether to optimize.

Optimal stripe size really depends on what your I/O pattern for the most important (or most common) cases. You can crudely approximate this by running those tasks while watching Activity Monitor. In Activity Monitor select "Disk" and divide "Data read/sec" by "Reads in/sec". E.g, 100 megabytes/sec divided by 100 reads/sec = 1 megabyte I/O size. It fluctuates a lot -- you just want a rough number. Ditto for writes, but typically reads predominate so are more important.

You can also do the same for "data read" divided by "reads in", which is the total amount since last reboot. However unless you rebooted right before the test, you don't know how much of that cumulative activity was important. This is very rough, and the optimal stripe size for one I/O pattern will be
sub-optimal for others.

Nowadays still image and video files are very large -- a single raw photo from a DSLR can be 60 megabytes. However the key issue what I/O size for your software uses. Also what activities are most time-consuming and I/O-bound. That varies with your workflow. E.g. many activities are already CPU-limited, not I/O-limited. Further improving I/O won't help in those cases.

Even though RAID5 has some redundancy, you still need to backup everything on another drive or array.

Please Log in or Create an account to join the conversation.

Needed: How to Setup External Hard Drives For FCPX 18 Aug 2015 13:57 #67555

  • shanesangle
  • shanesangle's Avatar
  • Offline
  • User is blocked
  • User is blocked
  • Posts: 12
  • Thank you received: 0
Thank you, Joema.

That is really useful - I appreciate you taking the time to respond to my question.

I will try 512kb stripe sizes.

I haven't done enough editing to test I/O speeds yet. But I will attempt to do so shortly.

Can I back up the RAID using a single ext HD (say a 4 TB)? I guess this 4TB ext HD would be 'backed up' by doing intermittent 'clones' of all data on the RAID; or are there Time Machine type backups/apps which can control these?

Thanks again, Joema.
Shane

Please Log in or Create an account to join the conversation.

Needed: How to Setup External Hard Drives For FCPX 18 Aug 2015 14:23 #67556

  • joema
  • joema's Avatar
  • Offline
  • Platinum Boarder
  • Platinum Boarder
  • Posts: 1666
  • Karma: 27
  • Thank you received: 361
An R4 with 8TB raw capacity has about 6TB formatted capacity in RAID5. You can back that up to a smaller drive using either Time Machine or Carbon Copy Cloner or other software. However it is much better to have a backup drive at least the size of your data drive plus boot drive. This is because incremental backups can take more space.

I back my R4 continuously via Time Machine to an 8TB RAID0 G-RAID: amzn.com/B00846Z4YY
Also every day or two to a separate 8TB WD Mybook Duo RAID0 via Carbon Copy: amzn.com/B00F0JXDB4
Plus other selected backups on a variety of other drives.

Both Time Machine and Carbon Copy do incremental backups but Time Machine is very easy to use and runs continuously. However I would not trust critical data to a single backup method.

Please Log in or Create an account to join the conversation.

Needed: How to Setup External Hard Drives For FCPX 18 Aug 2015 22:18 #67573

  • shanesangle
  • shanesangle's Avatar
  • Offline
  • User is blocked
  • User is blocked
  • Posts: 12
  • Thank you received: 0
Thanks Joema,

Your advice has been invaluable to me.

Please Log in or Create an account to join the conversation.

  • Page:
  • 1