fcp.co logo transparent
fcp clapperboard
Welcome, Guest
Username: Password: Remember me
25 Jan 2021
New boarders will have their posts moderated - Don't worry if you cannot see your post immediately.
Read More...

TOPIC:

Media & Cache on shared storage network 10 Feb 2020 17:58 #104518

oldsoul-

I'd encourage you to reach out to SNS as it appears they have a fix until Apple can address this at the OS level.

I tried it on Friday, and everything is working and looking good for our Catalina based machines.

Hope this helps!

Please Log in to join the conversation.

Media & Cache on shared storage network 10 Feb 2020 19:43 #104520

  • oldsoul
  • oldsoul's Avatar Topic Author
  • Offline
  • Senior Member
  • Senior Member
  • Posts: 44
  • Karma: 1
  • Thank you received: 1
Thanks Jeremy G. Thats great news. I've been working with SNS since the beginning and have been diligent about trying to solve this. They are working on a Mojave patch now and will let me know. Fingers crossed.

Please Log in to join the conversation.

Media & Cache on shared storage network 11 Feb 2020 19:15 #104541

Actually, it was a very dumb thing they changed without letting anyone know. There is an easy fix for this, we have tested it with different clients.

- Ronny

Please Log in to join the conversation.

Media & Cache on shared storage network 11 Feb 2020 23:51 #104550

Wow, thanks to everyone for troubleshooting this! I thought we were all going crazy. Every once in a while we would have a disconnection, or the library refused to save. Our storage is connected through SMB. I THINK we mitigated the problem somewhat by pointing to the cache to a different subfolder whenever we switched machines.

It's crazy because Apple has broken SMB several times over the years, and they often don't communicate how they're changing the implementation. Whoever's in charge of this needs to be fired.

Please Log in to join the conversation.

Media & Cache on shared storage network 12 Feb 2020 15:46 #104567

Can you specify the "dumb thing" that they changed? We have a QNAP SMB server and I would like to know if there is a way for us to deal with the issue while waiting on Apple.

Please Log in to join the conversation.

Media & Cache on shared storage network 13 Feb 2020 12:44 #104581

I am not an IT engineer, I repeat what our techies are saying. But it has something to do with port usage and netbios. Just like SNS, our support team has released a patch for our clients. I am sure that Qnap will do the same soon.

- Ronny

Please Log in to join the conversation.

Media & Cache on shared storage network 13 Feb 2020 13:29 #104584

We had the Mojave machines fixed yesterday and there’s even improvement there, even though those machines weren’t dropping connections.

Please Log in to join the conversation.

Goo 13 Feb 2020 13:31 #104585

Good to hear that, Jeremy! It's actually the same fix, but with a little tweak.

- Ronny

Please Log in to join the conversation.

Media & Cache on shared storage network 19 Feb 2020 22:09 #104768

It would be greatly appreciated if someone who is working on solutions could actually publish specific technical details on both the issues introduced in macOS Catalina and the current work arounds being implemented by teams such as yours and other teams like SNS. I keep seeing this issue posted everywhere online, but no one is posting anything helpful for people like myself that have to try and DIY fix this kind of an issue. If you have any ability to get a more technical explanation on potential fixes, it would be so incredibly appreciated. Thanks.

Please Log in to join the conversation.

Media & Cache on shared storage network 20 Feb 2020 13:08 #104780

For us, it was a machine to machine fix.

I would suggest getting in touch with QNAP to see if they can help you out.

Please Log in to join the conversation.

Media & Cache on shared storage network 27 Feb 2020 02:38 #104969

After contacting Qnap they've confirmed they haven't heard this, but a ticket has been filed. Looking at the available information it appears that this has nothing to do with Qnap, Lumaforge, or SNS. They just happened to have figured it out and made changes to MacOS itself rather than updating the firmware on the NAS. While this is nice for those customers it leaves everyone else out in the cold until Apple acknowledges and fixes the problem. If this is the case there has to be a way to get this working without having to be an SNS or Lumaforge customer. They have so far not published any info on how the fixes work.

Please Log in to join the conversation.

Media & Cache on shared storage network 27 Feb 2020 10:05 #104975

Hi Grant, welcome to the forums.

The issue only is apparent when working with FCP X and Catalina under heavy workloads, which I don't always expect from people using a small desktop NAS. So it's not really surprising to hear that QNAP is not aware of these issues.

All professional NAS manufacturers have been working on a fix. LumaForge has released an update to the Jellyfish Connect App that manages the interaction between the workstations and the server and fixes the Catalina issues. Such updates take engineering time and budget, and they are part of the support that is included in the price that clients pay for their systems. So it's quite obvious that no-one will publish any information about this. I'm sure that QNAP will release their own fix soon if they get more people reporting these issues.

- Ronny

Please Log in to join the conversation.

Media & Cache on shared storage network 02 Mar 2020 19:43 #105111

Here's a possible fix for folks like myself that don't have the same support as other NAS solutions like Lumaforge and SNS.

I have literally spent an insane amount of time researching ways to improve network performance in macOS Catalina and I have finally stumbled across a solution. I have documented it on Apple's community website.

discussions.apple.com/thread/251102946?page=1

It involves a built in macOS feature called "Server Performance Mode". It seems to be that macOS Catalina default system parameters for networking are not tuned for high demand network performance. That being said, it seems that at some point in time as far back as OS X Mountain Lion some engineers at Apple created a feature called “Server Performance Mode”. Imagine that, a feature in macOS literally named after creating a network performance mode. This feature is documented on Apple.com

Server Performance Mode
support.apple.com/en-us/HT202528

The description reads:
Performance mode changes the system parameters of your Mac. These changes take better advantage of your hardware for demanding server applications. A Mac that needs to run high-performance services can turn on performance mode to dedicate additional system resources for server applications.

Implementing this feature on my brand new Mac Pro (2019) connected to a QNAP system fixed issues with Final Cut Pro X freezing and disconnecting SMB network volume shares. It also increased my throughput performance from 30%-50%. My machine almost reaches the theoretical max throughput of 10GbE. I have also tested this on older machines running Mojave and they also gained performance improvements. I have also tested older "trash can" Mac Pro's upgraded to Catalina that were previously also crashing and disconnecting SMB shares and they too are now working. I can't promise that this will fix everybody's issues, but I can tell you that there is a good chance it will help your overall network performance especially if you are using a high demanding network application.

I just can't believe this feature exists and is yet so hard to find, It has apparently been part of macOS for a long time. I have been talking with this issue with Apple directly the entire time leading up to my discovery and they refused to help in anyway or admit that anything was missed configured in macOS Catalina. So to say I have been a frustrated Apple customer is an understatement. Their FCPX team and engineers should be more aware of built in macOS features that help and affect network performance.

I hope this helps someone.

Please Log in to join the conversation.

Media & Cache on shared storage network 02 Mar 2020 22:22 #105115

  • joema
  • joema's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Posts: 1845
  • Karma: 27
  • Thank you received: 419

afite wrote: ...Implementing this feature on my brand new Mac Pro (2019) connected to a QNAP system fixed issues with Final Cut Pro X freezing and disconnecting SMB network volume shares. It also increased my throughput performance from 30%-50%. My machine almost reaches the theoretical max throughput of 10GbE....I just can't believe this feature exists and is yet so hard to find, It has apparently been part of macOS for a long time. I have been talking with this issue with Apple directly the entire time leading up to my discovery and they refused to help in anyway or admit that anything was missed configured in macOS Catalina. So to say I have been a frustrated Apple customer is an understatement. Their FCPX team and engineers should be more aware of built in macOS features that help and affect network performance....


Thanks for that info. The wording implies Server Performance Mode is intended for server-side scenarios such as database servers running on MacOS. The current Catalina NAS problem reverses that where the NAS is the server and MacOS is the client. However if it fixes your situation, that is great. This is a serious issue. Apple does not provide mission-critical "hot fixes", and there are valid pros/cons to that practice. However this leaves customers fending for themselves.

Server Performance Mode is a documented feature and if it helps someone suffering from the Catalina NAS problem, they can try it.

This post implies that disabling System Integrity Protection is needed to enable Server Performance Mode: www.codingmerc.com/blog/enable-server-pe...-on-os-x-el-capitan/

Afterward you'd want to enable SIP, as that protects against things like the Chrome Keystone bug which rendered many Macs non-bootable: arstechnica.com/information-technology/2...d-macs-from-booting/

Please Log in to join the conversation.

Last edit: by joema.

Media & Cache on shared storage network 02 Mar 2020 22:55 #105116

I can confirm the Server Performance Mode does indeed work. Not sure why you think System Integrity Protection needs to be disabled as I tried this on multiple machines without having to disable SIP. Googling the feature makes it seem like SIP must be disabled, but that has not been my personal experience. Only a reboot is needed. FCP stability and general network speed to the network shares are greatly increased and are far more stable when compared to the default settings.

If both SNS and Lumaforge have documented that this is a MacOS problem but have neglected to release any technical info on their solutions, this solution remains the only publicly available option that actually works. It also happens to be a long standing native feature of MacOS. The same google search reveals this isn't dangerous and doesn't make the client computer a "server" in the classic sense, instead just changing the tolerances for simultaneous connections, packet size, and memory usage. If you're running an old mac this might not be a good idea. If you're running a Mac Pro for the purpose of, oh I don't know, editing in FCP with a NAS, the hardware resources far exceed these thresholds.

If anyone feels uncomfortable enabling this feature, by all means wait for Apple to fix it, though I doubt that will happen given their response mentioned above.

Please Log in to join the conversation.

Media & Cache on shared storage network 03 Mar 2020 14:50 #105147

@awkward_eagle
Glad to hear this fix is working for someone else too.

@joema
I was not aware of the potential need to disable SIP. Thanks for the tip as it's good to know that the situation may come up. For whatever reason, my systems both running Catalina and Mojave did not require the SIP disabling in order to enable Server Performance Mode. I would be curious to know which hardware and OS configurations require it. The link you sent makes it look like it was required for El Capitan.

Please Log in to join the conversation.

Media & Cache on shared storage network 03 Mar 2020 15:23 #105153

  • joema
  • joema's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Posts: 1845
  • Karma: 27
  • Thank you received: 419

afite wrote: ...I was not aware of the potential need to disable SIP...my systems both running Catalina and Mojave did not require the SIP disabling in order to enable Server Performance Mode....The link you sent makes it look like it was required for El Capitan.


The link I sent said El Capitan and later. However this may have changed. The Apple article gives a procedure using a terminal NVRAM command to set Server Performance Mode, then to interrogate it to ensure it is enabled. If people are using that and it reports enabled, that's good, assuming it doesn't cause any other issues.

More info on what Server Performance Mode does:

apple.stackexchange.com/questions/264958...actually-do-on-macos

Please Log in to join the conversation.

Last edit: by joema.

Media & Cache on shared storage network 03 Mar 2020 20:00 #105156

As a side note, if you need to temporarily disable System Integrity Protection in order to enable Server Performance Mode, SIP can be reenabled and the fix will remain in place. The only time you'd need to reenable Server Performance Mode is if you do an nvram reset.

Please Log in to join the conversation.

Media & Cache on shared storage network 05 Aug 2020 16:08 #109131

afite wrote: Here's a possible fix for folks like myself that don't have the same support as other NAS solutions like Lumaforge and SNS...

I have literally spent an insane amount of time researching ways to improve network performance in macOS Catalina and I have finally stumbled across a solution. I have documented it on Apple's community website.

discussions.apple.com/thread/251102946?page=1

Thanks so much for sharing.

I did some digging, to see what Lumaforge is doing for SMB with their so called "Magic" in their "Jellyfish Connect" app. As far as I can see, they simply change the following values in ~/Library/Preferences/nsmb.conf (can also be set for all users in /etc/nsmb.conf).
[default]
signing_required=no
protocol_vers_map=6
port445=netbios_only
I believe that port445=netbios_only setting is what Ronny was mentioning?

You can find more info about these settings in man nsmb.conf via Terminal.
port445 - How to use SMB TCP/UDP ports

"How to use SMB TCP/UDP ports" can be one of:

both - Attempt to connect via port 445. If that is unsuccessful, try to connect via NetBIOS.

netbios_only - Do not attempt to connect via port 445.

no_netbios - Attempt to connect via port 445. If that is unsuccessful, do not try to connect via NetBIOS.

"Bitmap of SMB Versions that are enabled" can be one of:

7 == 0111 - SMB 1/2/3 should be enabled
6 == 0110 -SMB 2/3 should be enabled
4 == 0100 - SMB 3 should be enabled

"Bitmap of SMB Versions that have signing required" can be one of:
7  Signing required for SMB 1/2/3.
6  Signing required for SMB 2/3.
4  Signing required for SMB 3.

They also set the following values for NFS and a few specific settings for those with Myricom 10 GbE adapters.

cat /etc/nfs.conf
# nfs.conf: The NFS Configuration File
# LumaForge Optimized
nfs.client.access_cache_timeout=30
nfs.client.nfsiod_thread_max=15
nfs.client.access_for_getattr=1
nfs.client.allow_async=1
nfs.client.mount.options=retrycnt=10,rwsize=65536,dsize=16384,readahead=128,rw,async
nfs.client.nextdowndelay=15
nfs.client.statfs_rate_limit=5
# End LumaForge Optimized
/etc/myri_settings.conf
net.myri10ge.en*.lro_cnt=8
net.myri10ge.en*.intr_coal_delay=5
They also check the NIC, and set 10 GbE NICs to Jumbo frames, if found. If it's 1 GbE, MTU is set to auto (1500).

I really don't understand why there is so much "secrecy" as to what different vendors are doing to solve the network performance issues in macOS. Can't we all just collaborate on getting the best performance from the macOS network stack, without hiding information from each other? What's the point?

ATTO on the other hand has created a free tuning tool for ATTO Network cards, called ATTO360. The actual settings they set for the different sysctl tuning profiles are as follows for 10 GbE FastFrame NICs, while there might be different settings for their new FastFrame3 10/25/50/100 NICs.

/etc/sysctl.conf
Writing settings in this file makes settings stick on reboot. If you want to see them "on-the-fly", you simply type the following command for each line. E.g.:
sudo sysctl -w net.inet.tcp.autorcvbufmax=2097152
High Throughput and Multi Stream Throughput
net.inet.tcp.autorcvbufmax=2097152
net.inet.tcp.autosndbufmax=2097152
net.inet.tcp.delayed_ack=0
net.inet.tcp.lro=1
net.inet.tcp.tso=1
net.inet.tcp.win_scale_factor=8
net.link.generic.system.hwcksum_rx=1
net.link.generic.system.hwcksum_tx=1
Low Latency
net.inet.tcp.autorcvbufmax=1048576
net.inet.tcp.autosndbufmax=1048576
net.inet.tcp.delayed_ack=0
net.inet.tcp.lro=0
net.inet.tcp.tso=4
net.inet.tcp.win_scale_factor=8
net.link.generic.system.hwcksum_rx=1
net.link.generic.system.hwcksum_tx=1
I'm actually seeing a huge difference in responsiveness wtih SMB read operations when net.inet.tcp.delayed_ack is set to 0, compared to the default value of 3 in macOS 10.15 Catalina.

ATTO is also explaining the different options in a PDF, which is pretty nice!
www.atto.com/pdfs/PRMA-0495-000.pdf

LRO – Large Receive Offload is a technique for increasing inbound throughput of high-bandwidth network conditions by reducing CPU overhead

TSO – TCP segmentation Offload is a technique for increasing outbound throughput of high-bandwidth network communications by reducing CPU overhead.


This partly explains why they set different values for "net.inet.tcp.lro" and "net.inet.tcp.tso", depending on whether you need extra Low Latency or extra High Throughput.

As always, be careful about what you change, and make a backup of your previous settings before changing.

Some tips regarding troubleshooting of SMB connectivity issues in macOS
In case anybody needs to troubleshoot SMB in macOS 10.3.6 and newer, these commands might reveal some interesting details/problems.

1. Enable SMB debug logging:
sudo sysctl -w net.smb.fs.loglevel=99
2. Start streaming log messages:
log stream --level debug --predicate 'senderImagePath contains "smb"'
3. Reproduce issues

You can also show log messages after testing with the following command:
log show --debug --predicate 'senderImagePath contains "smb"' --last 1h
When done testing, disable SMB debug logging in the end:
sudo sysctl -w net.smb.fs.loglevel=0

In some cases, you might need to capture some tcp packets as well, for further Wireshark analysis.

The commands above recently revealed some network memory (mbuf) issues with some 10 GbE macOS clients reading 2K dpx image sequences in Blackmagic DaVinci Resolve. The smb debug logs clearly showed that the SMB connection failed, and indicated sock_receivembuf being the root cause of the broken connection.
2019-03-06 14:41:34.564 Df kernel[0:398cd] (smbfs) nbssn_recv: Breaking connection, sock_receivembuf blocked for 20
2019-03-06 14:41:34.564 Df kernel[0:398cd] (smbfs) smb_iod_reconnect: Starting reconnect with server
2019-03-06 14:41:34.564 Df kernel[0:3992d] (smbfs) smb2_smb_parse_change_notify: smb_rq_reply failed 60
2019-03-06 14:41:34.564 Df kernel[0:398cd] (smbfs) tcp_connect: nbp_rcvbuf = 7405568 nbp_sndbuf = 7405568
2019-03-06 14:41:40.566 Df kernel[0:398cd] (smbfs) smbfs_down: Share not responding...
Simply increasing the mbuf to 512 MB solved the issue.

Running the Terminal command netstat -mm will also help tell you if you have any requests for memory delayed, which is a clear sign of “not enough memory”.

To change the mbuf value in macOS, boot into recovery mode (cmd+R during boot) and run the following Terminal command to increase it to 512 MB: nvram boot-args="ncl=262144"

The tip about increasing the mbuf value in macOS is explained in more detail in a recent Dell EMC Isilon macOS tuning doc: www.dellemc.com/resources/en-us/asset/wh...nce_optimization.pdf


Lastly, I'm also sharing some additional info regarding /etc/nsmb.conf, that I got from somebody in the Dell Isilon team a couple of years ago. Keep in mind that this was for macOS 10.13.6. However, it's still useful to see what the different options mean.
#
# /etc/nsmb.conf - Global Config
# ~/Library/Preferences/nsmb.conf - User Config, Overrides Global Config
# No reboot required, just unmount and remount
# man nsmb.conf - to view localized options
#
#
# Aug 28, 2018
# For macOS X 10.13 High Sierra
# Tested on macOS X 10.13.6 High Sierra
#
[default]
# Avoids any Kerberos or NTLM questions from the mac and assumes non-certed creds (standard in most cases)
# but if you have Kerb or NTLS you may want to set this to that - Speeds up mounting.
minauth=none
 
# NetBOIS on isilon is no good, this forces the mac to net even attempt NB and stricktly use DNS
port445=no_netbios
 
# Allows the mac to have synchronous R or W access to the same file within different SMB calls.
# Faster performance in some cases.
streams=yes
 
# Soft mounts, same concept as NFS soft mounts,
# offloads application hangs to the OS and decreases
# the chance of an application hang or crash because
# of some but below layer 4, i.e. lacp break, alt route, IO stall or busy, etc. - application stability
soft
 
# Change notifications are off. This is a lengthy one to explain but i'll try to summarize,
# macOS is not good with open file handles over SMB.
# This means when a user has been doing a bunch of work that day on the storage
# a bunch of lingering file handles will somewhat stay open,
# change notify is a part of the SMB stack which is a background communication
# between the mac and storage which will be chattier and chattier as the day goes on.
# This doesnt stop open file handles but it does stop the chattiness on the line
notify_off=yes
 
# This is the bit mode that forces SMB3 only connections and SIGNIFICANTLY decreases mount time from seconds to almost zero.
# If you have other non-SMB3 storage your users need to use another bit mode. you can find that in the man.
protocol_vers_map=4
 
# Already default now in 10.13 but i like it there in case Apple changes something again, which they do.
signing_required=no
 
# Checks that the negotition is infact the SMB level that is being requested
# by sending some test data that only the protocol can support.
# Slows things down, trust us, its SMB3.
validate_neg_off=yes
 
# Makes protocol stalls more apparent therefor making identifiable problems more apparent.
max_resp_timeout=1s
 
# How many simul requests can macOS make to ask find what is in an open finder window.
dir_cache_async_cnt=1000
 
# 0 means no max
dir_cache_max=0s
 
# 0 means no min
dir_cache_min=0s
 
# This is the ability to keep the cache and refer back to it.
# Having that enabled does not show any updates to the finder (finder refresh related).
# If the dir_cache is turned off then you make the request to fill the
# "temp" cache every time you jump into a folder instead of referring to a premade dir-cache.
dir_cache_off=yes
 
# Unknown, but helps some applications save/overwrite files ?
file_ids_off=yes

Please everybody, let's keep sharing our findings and stop hiding common knowledge like this.

Please Log in to join the conversation.

Media & Cache on shared storage network 30 Jan 2021 06:51 #112435

Maybe this is the reason for the disconnection of FCPX and SMB
developer.apple.com/support/kernel-extensions/

Please Log in to join the conversation.