Intel Xeon Gold 6146 12-Core/24-Thread Processor Review

Introduction Desktop processors are great, for desktop usage. When you need to run your servers and high-end workstations, then you need something like Intel's Xeon Scalable Processors. Today I'm taking a closer look at the Xeon Gold 6146 (product link) from Intel, which is a 12-core/24-thread processor with a 3.2GHz base clock. Intel Xeon Scalable Processors These are Intel's latest iteration of Xeon processors which use the Skylake architecture. The official rating is 10% IPC gain over the predecessor, but that isn't what sets the new chips apart. The processors replace the now ageing Broadwell Xeon platform with up to 28 cores and 56 threads. In addition to the host of regular Skylake optimisations, Intel is throwing in AVX 512 with 2 FMA per core. This should double floating point performance in the most optimal scenario. AVX 512 will be a Skylake-SP exclusive until Cannon Lake at the earliest. Starting with Skylake-SP, the chip giant is moving to a new multi-core paradigm and calling the new design Mesh. Mesh has the entire multi-core architecture revamped to enhance flexibility. This is the system Intel is using to connect all 28 cores in Skylake-SP. With Broadwell-EX, Intel connected each core to a ring bus which facilitated inter-core communication. Depending on the specific cores, the ring bus could be the bottleneck and larger cores required two ring busses. With Mesh, each logic block is connected using several interconnects multiple units. Due to the extra connections, inter-core communication is improved. Another benefit of the design is the improved scaling, hence the name. Adding a new core is as simple as plugging in a few interconnects. Chipset and Socket This generation of Xeon processors uses the LGA3647 socket which is the same used by Xeon Phi x200 (Knights Landing). The socket is quite large, as the name suggests, and it requires a new generation of coolers. At the time of this review, the processor generation is so new that the hardest part of this review was to get hold of a compatible cooler. The socket supports 6-channel memory controller, non-volatile 3D XPoint memory DIMMs, and Intel UltraPath Interconnect (UPI). Both the CPU and the socket are almost new, we've only seen them once before. That's a lot of connection pins, but when we consider the amount of processing power and lanes available, it's no surprise. Some of the pins are used for the stackable architecture too. The chipsets used are the Intel C620-series, more specific the C621, C622, C624, C625, C626, C627, and C628. The number of differences isn't high, but not unimportant. The main differences are the Intel QuickAssist technologies supported, but also the Ethernet ports. Xeon Gold 6146 Processor Intel's Xeon Gold 6146 is from the 6XXX family which is the 2nd highest tier. Only the 8xxx Platinum series comes with a higher performance and more features. Among the highlights is a significant workload-optimized performance improvement for general purpose computing. Storage and network workloads are another highlight while we get top memory support and enhanced Intel AVX-512. The scalability of the series goes up to 4 sockets for high-end performance systems. The Xeon Gold 6146 is a server processor built on a 14nm process. It features 12 cores and 24 threads, running at a base clock of 3.2GHz. The single-core boost max is rated at 4.2GHz turbo. Inside the chip, you'll find 24.75MB L3 cache packed in a 165W TDP package. Features Speed Shift Technology Turbo Boost Technology 2.0 vPro Technology Hyper-Threading Technology Virtualization Technology (VT-x) Virtualization Technology for Directed I/O (VT-d) VT-x with Extended Page Tables (EPT) TSX-NI Instruction Set Extensions: SSE4.2, AVX, AVX2, AVX-512 2x AVX-512 FMA Units Enhanced Intel SpeedStep® Technology Intel Volume Management Device (VMD) AES New Instructions Intel Trusted Execution Technology Execute Disable Bit Intel Run Sure Technology Mode-based Execute Control (MBE) Packaging and Accessories Now, this is a server and not a retails processor, so I don't have any packaging to show. This kind of processors come without accessories as including would be a waste. They are used by system builders which already have everything they need. The processor, or multiple ones for that matter, are still well protected in the original tray. Intel Xeon Scalable Platform The way we process and utilise data has changed a lot in recent years. Our hardware has to follow the same trend in order to keep up with the demands for flexible compute, networking, and storage. Especially scaling is a thing that's very important, but as the name suggests, that's what Intel's Scalable Processors are good at. The exponential growth of data generation and consumption, the rapid expansion of cloud-scale computing, emerging 5G networks, and the extension of high-performance computing (HPC) and artificial intelligence (AI) into new usages require that today's data centres and networks urgently evolve. It is either that or be left behind in a highly competitive environment. The platform provides the foundation for a powerful data centre setup. With its new Mesh Architecture and widely expanded resources to its hardware-accelerating and newly integrated technologies, the platform enables a new level of consistent, pervasive, and breakthrough performance. Lower Your Total Cost of Ownership A new generation of hardware usually comes with performance and efficiency improvements. That is also the case here with this new Intel series. The Xeon Scalable platform is designed to lead to an improved total cost of ownership as well as higher productivity for users. On an overall level, the TCO can be lowered up to 65 percent with these processors. Factors within that calculation are software and OS licensing fees, acquisition, maintenance, and infrastructure costs. Virtualization is one of the places where the Intel Xeon Scalable platform shines. That is no surprise and it is an important factor in the enterprise market. More and more users take advantage of virtual machines, and the Scalable platform can run 4.2 times more VM per server when compared to earlier generations. Foundational Enhancements You already know your current platform and you want to know why you should upgrade to the latest generation? For this, I've added a list of changes and additions in bullet-points. It will hopefully allow you to find out whether the upgrade is worth it or not for your setup. Higher per-core performance. Up to 28 cores, delivering high performance and scalability for compute-intensive workloads across compute, storage, and network usages. Greater memory bandwidth and capacity. 50% increased memory bandwidth and capacity. Six memory channels versus four memory channels of the previous generation. Expanded I/O. 48 lanes of PCIe 3.0 bandwidth and throughput for I/O-intensive workloads. Intel Ultra Path Interconnect (Intel UPI): Up to three Intel UPI channels increase the scalability of the platform to as many as eight sockets, as well as improves inter-CPU bandwidth for I/O-intensive workloads versus the previous generation. Intel Advanced Vector Extensions 512 (Intel AVX-512). With double the flops per clock cycle compared to previous generation Intel Advanced Vector Extensions 2 (Intel AVX2), Intel AVX-512 boosts performance, and throughput for the most demanding computational tasks in applications, such as modelling and simulation, data analytics and machine learning, data compression, visualization, and digital content creation. Security without compromise. Near-zero encryption overhead enables higher performance on all secure data transactions. First-time Platform Integrations There are first-time features too, things that the previous generation doesn't have. Integrated Intel Omni-Path Architecture (Intel OPA) Host Fabric Interface in the CPU. End-to-end high-bandwidth, low-latency fabric that optimizes performance and eases deployment of HPC clusters by eliminating the need for a discrete host fabric interface card. Integrated Intel QuickAssist Technology (Intel QAT). Chipset-based hardware acceleration for growing compression and cryptographic workloads for greater efficiency while delivering enhanced data transport and protection across server, storage, and network infrastructure. Integrated Intel Ethernet into the chipset with scalable iWARP RDMA. Provides up to four 10 Gbps Ethernet ports for high data throughput and low-latency. Ideal for software-defined storage solutions, NVM Express over Fabric solutions, and virtual machine migrations. Enhanced Platform Trust Data and platform reliability, as well as protection, are key concerns for enterprises. The new platform also helps you in this area. Enhanced Intel Run Sure Technology. New enhancements deliver advanced Reliability, Availability, and Serviceability (RAS) and server uptime for a company’s most critical workloads. Hardware-assisted capabilities, including enhanced MCA and recovery and adaptive multi-device error correction, diagnose, and recover from previously fatal errors. And, they help ensure data integrity within the memory subsystem. Intel Key Protection Technology (Intel KPT) with Integrated Intel QAT and Intel Platform Trust Technology (Intel PTT). Deliver hardware-enhanced platform security by providing efficient key and data protection at rest, in-use, and in-flight. Intel Trusted Execution Technology (Intel TXT) with One-Touch Activation. Enhanced platform security while providing simplified and scalable deployment for Intel TXT. You Want to Know More? There are a whole lot more details of which you can read up on, on Intel's product brief and product page. You can even download the whole thing as a PDF from there are read it when and where you want. The Setup and Test Methodology It is not an easy job to come up with fair tests for a processor. After all, you can't run a CPU without a lot of other hardware and all that hardware will affect the results. Still, I've tried to dig out as many tests as possible which show what the processor is capable. Hopefully, it will show us where the strengths and weaknesses are located. The operating system used will be Windows Server 2012 R2 which is the same that we used with previous server motherboard reviews. It should also be noted that we're only using a dedicated GPU for monitor out. By eliminating the high-performance graphics card, we are able to compare the pure performance. Additional Hardware Motherboard: Supermicro X11SPI-TF Cooler: Supermicro 4U Active CPU Heat Sink (SNK-P0070APS4) RAM: Crucial DDR4 2133 MHz (8x 16GB) OS Drive: Toshiba OCZ VX500 512GB SATA3 GPU: Sapphire Ultimate R7 250 1G D5 Calculation Tests Having the CPU calculate is an easy way to show how well it does. The quicker the test is performed, the better the performance. For these tests, I've picked Super Pi and wPrime. Both are applications familiar to most and the results are easily comparable. SuperPi Mod 1.9 WP wPrime 2.10 Benchmark Suites Suites are popular too as they offer up a whole lot of results with very little effort. We'll be using AIDA64, GeekBench4, and PerformanceTest 8 here. AIDA64 Engineering Edition GeekBench 4 PerformanceTest 8 Rendering Tests Rendering tests are mostly for handled by your graphics card, but not solely. While these aren't exactly intended areas of operation, they do offer us comparable figures between our reviews. We'll be using HandBrake for H.265 conversion of 4K video as well as Cinebench 11.5 and 15 here. Cinebench R11.5 Cinebench R15.0 HandBrake 4 SiSoft Sandra and SpecWPC SiSoftware's Sandra is technically a benchmark suite too, but it offers a lot more details. It tests more and different things, so it deserves its own category. SpecWPC is also a whole set of tests and an industry standard. The results are directly comparable with other reviews which makes it great. Most professionals won't look at anything else. SiSoftware Sandra Platinum SP2/SP4 SpecWPC 1.2 Memory Performance Memory is another thing which is affected by a processors capabilities and performance. For that reason, I've chosen to include those test results too. AIDA64 Engineering Edition GeekBench 4 PerformanceTest 8 SiSoftware Sandra Platinum SP2/SP4 Review Notes It should be noted that some charts don't make a whole lot of sense right now. Some tests are run for the first time and as such don't have any comparable values. As we test more of the new processors, we'll get more results and will update the charts. Calculation Tests SuperPi Super PI is a computer program that calculates pi to a specified number of digits after the decimal point—up to a maximum of 32 million. It uses Gauss–Legendre algorithm and is a Windows port of the program used by Yasumasa Kanada in 1995 to compute pi to 232 digits. wPrime wPrime is a leading multithreaded benchmark for x86 processors that tests your processor performance by calculating square roots with a recursive call of Newton's method for estimating functions, with f(x)=x2-k, where k is the number we're square rooting, until Sgn(f(x)/f'(x)) does not equal that of the previous iteration, starting with an estimation of k/2. It then uses an iterative calling of the estimation method a set amount of times to increase the accuracy of the results. Then it confirms that n(k)2=k to ensure the calculation was correct. This is repeated for all numbers from 1 to the requested maximum. Benchmark Suites AIDA64 Engineer AIDA64 Engineer provides several methods to measure system performance. These benchmarks are synthetic, i.e. the results show only the theoretical (maximum) performance of the system. In contrast to application tests, synthetic benchmarks do not tend to reflect the "real world" performance of the computer. These benchmarks provide a quick and easy comparison between computer states, e.g. when certain parameters (CPU clock speed, memory timings, etc.) change in system configuration. Geekbench 4 Geekbench 4 measures your system's power and tells you whether your computer is ready to roar. How strong is your mobile device or desktop computer? How will it perform when push comes to crunch? These are the questions that Geekbench can answer. PerformanceTest 8.0 PerformanceTest is a fast, easy to use software benchmarking tool that allows everybody to quickly assess the performance of their PC and compare it to a list of standard 'baseline' computer systems. With it, you can compare the performance of your machine, measure the effect of configuration changes and upgrades, and make objective, independent measurements on which to base your purchasing decision. Rendering Tests Cinebench R11.5 & R15.0 Cinebench is a real-world cross platform test suite that evaluates your computer's performance capabilities. It is a perfect tool to compare CPU and graphics performance across various systems and platforms, and best of all: It's completely free. HandBrake HandBrake is a tool for converting video from nearly any format to a selection of modern, widely supported codecs. There are three reasons to love this tool: It converts video from nearly any format, it is free and open-source, and it is available on Windows, Mac, and Linux. SiSoftware Sandra Platinum SiSoftware Sandra is a system analyser that includes benchmarking, testing and listing modules. It tries to go beyond other utilities to show you more of what is going on under the hood, so you draw comparisons at both a high and low-level in a single product. Arithmetic Cryptography Financial Analysis Multimedia SIMD Units Scientific Analysis SpecWPC 1.2 SPECwpc is the first benchmark to measure all key aspects of workstation performance based on diverse professional applications. The latest version is SPECwpc V1.2 (introduced on June 3, 2015), which extends performance measurement from physical to virtualised workstations. Results from SPECwpc V1.2 are comparable to those from SPEC V1.0.4, but not to any version before that. More than 30 workloads are included in SPECwpc V1.2 to test CPU, graphics, I/O and memory bandwidth. The tests are divided into the application categories listed below. Individual scores are generated for each test, and a composite score for each category is generated based on a reference machine, yielding a “bigger is better” result. The reference machine has the following configuration: x3430 processor, 8GB (4x2GB) memory, V4800 graphics, and SATA 7.2k rpm hard disk drive. Media and Entertainment Product Development Life Sciences Financial Services Energy General Operations Memory Bandwidth and Latency For the memory tests, we use the built-in memory benchmarks in AIDA64 Engineer, Geekbench 4, PerformanceTest, and SiSoftware Sandra. A wider array of tests will give us a wider array of results. AIDA64 Engineer AIDA64 Engineer provides several methods to measure system performance. These benchmarks are synthetic, i.e. the results show only the theoretical (maximum) performance of the system. GeekBench 4 Geekbench 4 measures your system's power and tells you whether your computer is ready to roar. PerformanceTest 8.0 PerformanceTest is a fast, easy to use software benchmarking tool that allows everybody to quickly assess the performance of their PC and compare it to a list of standard 'baseline' computer systems. Memory Read and Write Comparison Comparison SiSoftware Sandra Edition Sandra is a system analyser that includes benchmarking, testing and listing modules. It tries to go beyond other utilities to show you more of what is going on under the hood Memory Latency Final Thoughts and Pricing Pricing The official MSRP for Intel's Xeon Gold 6146 is $3286.00. At the time of review, I could not find this actual CPU listed on Amazon. There are pre-built servers with the CPU, so it is possible to get it. Other shops might also have it in stock, so check your favourite server-hardware retailer. Other SKUs from the Xeon Gold lineup are available. Conclusion The Intel Xeon Gold 6146 is no cheap processor and neither are the rest of the components you'll need. That said, it is well worth it in every way. At least if you need the performance it has to offer, and it does have a lot of that. The computing power is amazing on its own, but you don't have to stick with just one. With the scalability of the series, you can set up a system with two, four, or eight instead. While the performance of the Xeon Gold 6146 is great, there is a thing to keep in mind. This isn't the best of the new series. You can go even further, but only if you need to. We get a strong performance with the Gold 6146 beating dual-CPU setups of the previous generation in most scenarios. That's impressive! A job well-done! First-Time Features The new Xeon Scalable architecture is available with integrated Omni-Path Architecture. The Host Fabric Interface in the CPU offers end-to-end high-bandwidth and low-latency for an optimized performance and ease of deployment. It eliminates the need for a discrete host fabric interface card in your HPC clusters. We also get chipset-based hardware acceleration through Intel QAT. With it, you get a greater efficiency when working with compression and cryptographic workloads. Improved Features There are the obvious improvements when we compare to the predecessor such as higher per-core performance. The memory bandwidth and capacity were increased by 50% and we get a six-channel memory setup vs a four channel of the predecessor. There are more PCIe 3.0 lanes too. 48 lanes of PCIe 3.0 bandwidth allows you to run intensive I/O workloads. The Ultra Path Interconnect (UPI) offers up to three channels, allowing for an interconnection of up to eight sockets. General performance is increased through features too. Intel worked on the AVX-512 vector extension too which offers double the flops per clock cycle when compared to AVX2. You also get a near-zero encryption overhead, thus eliminating the choice between safety and performance. Pros Good Scalability Great Performance 48 PCIe 3.0 Lanes Six-Channel Memory Lower TCO when compared to predecessor Cons Difficult to get hold of at the time of review "Intel is back to show how server processors should be made. The Xeon Gold 6146 performs so well, it beats dual-core setups from the previous generation." Thank You, Intel, for this review sample.