Thanks so much for sharing.
I did some digging, to see what Lumaforge is doing for SMB with their so called "Magic" in their "Jellyfish Connect" app. As far as I can see, they simply change the following values in ~/Library/Preferences/nsmb.conf (can also be set for all users in /etc/nsmb.conf).
I believe that port445=netbios_only setting is what Ronny was mentioning?
[default] signing_required=no protocol_vers_map=6 port445=netbios_only
You can find more info about these settings in man nsmb.conf via Terminal.
port445 - How to use SMB TCP/UDP ports "How to use SMB TCP/UDP ports" can be one of: both - Attempt to connect via port 445. If that is unsuccessful, try to connect via NetBIOS. netbios_only - Do not attempt to connect via port 445. no_netbios - Attempt to connect via port 445. If that is unsuccessful, do not try to connect via NetBIOS. "Bitmap of SMB Versions that are enabled" can be one of: 7 == 0111 - SMB 1/2/3 should be enabled 6 == 0110 -SMB 2/3 should be enabled 4 == 0100 - SMB 3 should be enabled "Bitmap of SMB Versions that have signing required" can be one of: 7 Signing required for SMB 1/2/3. 6 Signing required for SMB 2/3. 4 Signing required for SMB 3.
They also set the following values for NFS and a few specific settings for those with Myricom 10 GbE adapters.
# nfs.conf: The NFS Configuration File # LumaForge Optimized nfs.client.access_cache_timeout=30 nfs.client.nfsiod_thread_max=15 nfs.client.access_for_getattr=1 nfs.client.allow_async=1 nfs.client.mount.options=retrycnt=10,rwsize=65536,dsize=16384,readahead=128,rw,async nfs.client.nextdowndelay=15 nfs.client.statfs_rate_limit=5 # End LumaForge Optimized
They also check the NIC, and set 10 GbE NICs to Jumbo frames, if found. If it's 1 GbE, MTU is set to auto (1500).
I really don't understand why there is so much "secrecy" as to what different vendors are doing to solve the network performance issues in macOS. Can't we all just collaborate on getting the best performance from the macOS network stack, without hiding information from each other? What's the point?
ATTO on the other hand has created a free tuning tool for ATTO Network cards, called ATTO360. The actual settings they set for the different sysctl tuning profiles are as follows for 10 GbE FastFrame NICs, while there might be different settings for their new FastFrame3 10/25/50/100 NICs.
Writing settings in this file makes settings stick on reboot. If you want to see them "on-the-fly", you simply type the following command for each line. E.g.:
High Throughput and Multi Stream Throughput
sudo sysctl -w net.inet.tcp.autorcvbufmax=2097152
net.inet.tcp.autorcvbufmax=2097152 net.inet.tcp.autosndbufmax=2097152 net.inet.tcp.delayed_ack=0 net.inet.tcp.lro=1 net.inet.tcp.tso=1 net.inet.tcp.win_scale_factor=8 net.link.generic.system.hwcksum_rx=1 net.link.generic.system.hwcksum_tx=1
I'm actually seeing a huge difference in responsiveness wtih SMB read operations when net.inet.tcp.delayed_ack is set to 0, compared to the default value of 3 in macOS 10.15 Catalina.
net.inet.tcp.autorcvbufmax=1048576 net.inet.tcp.autosndbufmax=1048576 net.inet.tcp.delayed_ack=0 net.inet.tcp.lro=0 net.inet.tcp.tso=4 net.inet.tcp.win_scale_factor=8 net.link.generic.system.hwcksum_rx=1 net.link.generic.system.hwcksum_tx=1
ATTO is also explaining the different options in a PDF, which is pretty nice!
LRO – Large Receive Offload is a technique for increasing inbound throughput of high-bandwidth network conditions by reducing CPU overhead
TSO – TCP segmentation Offload is a technique for increasing outbound throughput of high-bandwidth network communications by reducing CPU overhead.
This partly explains why they set different values for "net.inet.tcp.lro" and "net.inet.tcp.tso", depending on whether you need extra Low Latency or extra High Throughput.
As always, be careful about what you change, and make a backup of your previous settings before changing.
Some tips regarding troubleshooting of SMB connectivity issues in macOS
In case anybody needs to troubleshoot SMB in macOS 10.3.6 and newer, these commands might reveal some interesting details/problems.
1. Enable SMB debug logging:
2. Start streaming log messages:
sudo sysctl -w net.smb.fs.loglevel=99
3. Reproduce issues
log stream --level debug --predicate 'senderImagePath contains "smb"'
You can also show log messages after testing with the following command:
When done testing, disable SMB debug logging in the end:
log show --debug --predicate 'senderImagePath contains "smb"' --last 1h
sudo sysctl -w net.smb.fs.loglevel=0
In some cases, you might need to capture some tcp packets as well, for further Wireshark analysis.
The commands above recently revealed some network memory (mbuf) issues with some 10 GbE macOS clients reading 2K dpx image sequences in Blackmagic DaVinci Resolve. The smb debug logs clearly showed that the SMB connection failed, and indicated sock_receivembuf being the root cause of the broken connection.
Simply increasing the mbuf to 512 MB solved the issue.
2019-03-06 14:41:34.564 Df kernel[0:398cd] (smbfs) nbssn_recv: Breaking connection, sock_receivembuf blocked for 20 2019-03-06 14:41:34.564 Df kernel[0:398cd] (smbfs) smb_iod_reconnect: Starting reconnect with server 2019-03-06 14:41:34.564 Df kernel[0:3992d] (smbfs) smb2_smb_parse_change_notify: smb_rq_reply failed 60 2019-03-06 14:41:34.564 Df kernel[0:398cd] (smbfs) tcp_connect: nbp_rcvbuf = 7405568 nbp_sndbuf = 7405568 2019-03-06 14:41:40.566 Df kernel[0:398cd] (smbfs) smbfs_down: Share not responding...
Running the Terminal command netstat -mm will also help tell you if you have any requests for memory delayed, which is a clear sign of “not enough memory”.
To change the mbuf value in macOS, boot into recovery mode (cmd+R during boot) and run the following Terminal command to increase it to 512 MB: nvram boot-args="ncl=262144"
The tip about increasing the mbuf value in macOS is explained in more detail in a recent Dell EMC Isilon macOS tuning doc: www.dellemc.com/resources/en-us/asset/wh...nce_optimization.pdf
Lastly, I'm also sharing some additional info regarding /etc/nsmb.conf, that I got from somebody in the Dell Isilon team a couple of years ago. Keep in mind that this was for macOS 10.13.6. However, it's still useful to see what the different options mean.
# # /etc/nsmb.conf - Global Config # ~/Library/Preferences/nsmb.conf - User Config, Overrides Global Config # No reboot required, just unmount and remount # man nsmb.conf - to view localized options # # # Aug 28, 2018 # For macOS X 10.13 High Sierra # Tested on macOS X 10.13.6 High Sierra # [default] # Avoids any Kerberos or NTLM questions from the mac and assumes non-certed creds (standard in most cases) # but if you have Kerb or NTLS you may want to set this to that - Speeds up mounting. minauth=none # NetBOIS on isilon is no good, this forces the mac to net even attempt NB and stricktly use DNS port445=no_netbios # Allows the mac to have synchronous R or W access to the same file within different SMB calls. # Faster performance in some cases. streams=yes # Soft mounts, same concept as NFS soft mounts, # offloads application hangs to the OS and decreases # the chance of an application hang or crash because # of some but below layer 4, i.e. lacp break, alt route, IO stall or busy, etc. - application stability soft # Change notifications are off. This is a lengthy one to explain but i'll try to summarize, # macOS is not good with open file handles over SMB. # This means when a user has been doing a bunch of work that day on the storage # a bunch of lingering file handles will somewhat stay open, # change notify is a part of the SMB stack which is a background communication # between the mac and storage which will be chattier and chattier as the day goes on. # This doesnt stop open file handles but it does stop the chattiness on the line notify_off=yes # This is the bit mode that forces SMB3 only connections and SIGNIFICANTLY decreases mount time from seconds to almost zero. # If you have other non-SMB3 storage your users need to use another bit mode. you can find that in the man. protocol_vers_map=4 # Already default now in 10.13 but i like it there in case Apple changes something again, which they do. signing_required=no # Checks that the negotition is infact the SMB level that is being requested # by sending some test data that only the protocol can support. # Slows things down, trust us, its SMB3. validate_neg_off=yes # Makes protocol stalls more apparent therefor making identifiable problems more apparent. max_resp_timeout=1s # How many simul requests can macOS make to ask find what is in an open finder window. dir_cache_async_cnt=1000 # 0 means no max dir_cache_max=0s # 0 means no min dir_cache_min=0s # This is the ability to keep the cache and refer back to it. # Having that enabled does not show any updates to the finder (finder refresh related). # If the dir_cache is turned off then you make the request to fill the # "temp" cache every time you jump into a folder instead of referring to a premade dir-cache. dir_cache_off=yes # Unknown, but helps some applications save/overwrite files ? file_ids_off=yes
Please everybody, let's keep sharing our findings and stop hiding common knowledge like this.