View source for Hopper (microarchitecture)

{{Short description|GPU microarchitecture designed by Nvidia}}
{{Use mdy dates|date=December 2019}}
{{Infobox GPU microarchitecture
| name                   = Hopper
| image                  = 
| caption                = 
| alt                    = 
| launching              = 
| launched               = {{Start date and age|2022|09|20}}
| discontinued           = 
| soldby                 = 
| designfirm             = [[Nvidia]]
| manuf1                 = [[TSMC]]
| process                = TSMC [[5 nm process|N4]]
| codename               = 
<!------------------  Product Series  ------------------->
| products-desktop1      = 
| products-hedt1         = 
| products-server1       = [[Nvidia Tesla|Tesla]] H series
<!------------------  Specifications  ------------------->
| compute                = 
| slowest                = 
| slow-unit              = 
| fastest                = 
| fast-unit              = 
| shader-clock           = 
| l0-cache               = 
| l1-cache               = 256{{nbsp}}KB (per SM)
| l2-cache               = 50{{nbsp}}MB
| l3-cache               = 
| memory-support         = [[High Bandwidth Memory#HBM3|HBM3]]
| memory-clock           = 
| pcie-support           = [[PCI Express#PCI Express 5.0|PCI Express 5.0]]
<!------------------  Media  Engine ------------------->
| encode-codec           = 
| decode-codec           = 
| color-depth            = 
| encoders               = [[NVENC]]
| display-outputs        = 
<!------------------  History  ------------------->
| predecessor            = [[Ampere (microarchitecture)|Ampere]]
| variant                = [[Ada Lovelace (microarchitecture)|Ada Lovelace]] (consumer and professional)
| successor              = [[Blackwell (microarchitecture)|Blackwell]]
}}
[[File:NVIDIA H100 (极客湾Geekerwan) 025.png|thumb|4 Nvidia H100 GPUs]]
'''Hopper''' is a [[graphics processing unit]] (GPU) [[microarchitecture]] developed by [[Nvidia]]. It is designed for datacenters and is used alongside the [[Ada Lovelace (microarchitecture)|Lovelace]] microarchitecture. It is the latest generation of the line of products formerly branded as [[Nvidia Tesla]], now Nvidia Data Centre GPUs.

Named for computer scientist and [[United States Navy]] [[rear admiral]] [[Grace Hopper]], the Hopper architecture was leaked in November 2019 and officially revealed in March 2022. It improves upon its predecessors, the [[Turing (microarchitecture)|Turing]] and [[Ampere (microarchitecture)|Ampere]] microarchitectures, featuring a new [[Streaming Multiprocessor|streaming multiprocessor]], a faster memory subsystem, and a transformer acceleration engine.

==Architecture==

The Nvidia Hopper H100 GPU is implemented using the [[TSMC]] N4 process with 80 billion transistors. It consists of up to 144 [[Streaming Multiprocessor|streaming multiprocessor]]s.{{Sfn|Elster|Haugdahl|2022|p=4}} Due to the increased memory bandwidth provided by the SXM5 socket, the Nvidia Hopper H100 offers better performance when used in an SXM5 configuration than in the typical PCIe socket.{{Sfn|Nvidia|2023c|p=20}}

===Streaming multiprocessor===
The streaming multiprocessors for Hopper improve upon the [[Turing (microarchitecture)|Turing]] and [[Ampere (microarchitecture)|Ampere]] microarchitectures, although the maximum number of concurrent warps per streaming multiprocessor (SM) remains the same between the Ampere and Hopper architectures, 64.{{Sfn|Nvidia|2023b|p=9}} The Hopper architecture provides a Tensor Memory Accelerator (TMA), which supports bidirectional asynchronous memory transfer between shared memory and global memory.{{Sfn|Fujita|Yamaguchi|Kikuchi|Ichimura|2023|p=6}} Under TMA, applications may transfer up to 5D tensors. When writing from shared memory to global memory, elementwise reduction and bitwise operators may be used, avoiding registers and SM instructions while enabling users to write warp specialized codes. TMA is exposed through <code>cuda::memcpy_async</code>.<ref name=":1" />

When parallelizing applications, developers can use [[thread block (CUDA programming)|thread block]] clusters. Thread blocks may perform atomics in the shared memory of other thread blocks within its cluster, otherwise known as [[distributed shared memory]]. Distributed shared memory may be used by an SM simultaneously with [[L2 cache]]; when used to communicate data between SMs, this can utilize the combined bandwidth of distributed shared memory and L2. The maximum portable cluster size is 8, although the Nvidia Hopper H100 can support a cluster size of 16 by using the <code>cudaFuncAttributeNonPortableClusterSizeAllowed</code> function, potentially at the cost of reduced number of active blocks.{{Sfn|Nvidia|2023b|p=10}} With L2 multicasting and distributed shared memory, the required bandwidth for [[dynamic random-access memory]] read and writes is reduced.<ref name="NVIDIAVid">{{Cite AV media |people=Vishal Mehta |date=September 2022 |title=CUDA Programming Model for Hopper Architecture |language=en |url=https://www.nvidia.com/en-us/on-demand/session/gtcfall22-a41095/ |access-date=May 29, 2023 |location=Santa Clara |publisher=[[Nvidia]]}}</ref>

Hopper features improved [[single-precision floating-point format]] (FP32) throughput with twice as many FP32 operations per cycle per SM than its predecessor. Additionally, the Hopper architecture adds support for new instructions, including the [[Smith–Waterman algorithm]].{{Sfn|Nvidia|2023b|p=10}} Like Ampere, TensorFloat-32 (TF-32) arithmetic is supported. The mapping pattern for both architectures is identical.{{Sfn|Fujita|Yamaguchi|Kikuchi|Ichimura|2023|p=4}}

===Memory===
The Nvidia Hopper H100 supports [[HBM3]] and [[HBM2]]e memory up to 80 GB; the HBM3 memory system supports 3 TB/s, an increase of 50% over the Nvidia Ampere A100's 2 TB/s. Across the architecture, the L2 cache capacity and bandwidth were increased.{{Sfn|Nvidia|2023b|p=11}}

Hopper allows [[CUDA]] [[compute kernel]]s to utilize automatic inline compression, including in individual memory allocation, which allows accessing memory at higher bandwidth. This feature does not increase the amount of memory available to the application, because the data (and thus its [[Shannon's source coding theorem|compressibility]]) may be changed at any time. The compressor will automatically choose between several compression algorithms.{{Sfn|Nvidia|2023b|p=11}}

The Nvidia Hopper H100 increases the capacity of the combined L1 cache, texture cache, and shared memory to 256 KB. Like its predecessors, it combines L1 and texture caches into a unified cache designed to be a coalescing buffer. The attribute <code>cudaFuncAttributePreferredSharedMemoryCarveout</code> may be used to define the carveout of the L1 cache. Hopper introduces enhancements to [[NVLink]] through a new generation with faster overall communication bandwidth.{{Sfn|Nvidia|2023b|p=12}}

====Memory synchronization domains====
Some CUDA applications may experience interference when performing fence or flush operations due to memory ordering. Because the GPU cannot know which writes are guaranteed and which are visible by chance timing, it may wait on unnecessary memory operations, thus slowing down fence or flush operations. For example, when a kernel performs computations in GPU memory and a parallel kernel performs communications with a peer, the local kernel will flush its writes, resulting in slower NVLink or [[PCIe]] writes. In the Hopper architecture, the GPU can reduce the net cast through a fence operation.{{Sfn|Nvidia|2023a|p=44}}

===DPX instructions===
The Hopper architecture math [[application programming interface]] (API) exposes functions in the SM such as <code>__viaddmin_s16x2_relu</code>, which performs the per-[[halfword]] <math>max(min(a + b, c), 0)</math>. In the Smith–Waterman algorithm, <code>__vimax3_s16x2_relu</code> can be used, a three-way min or max followed by a clamp to zero.<ref>{{Cite web |url=https://developer.nvidia.com/blog/boosting-dynamic-programming-performance-using-nvidia-hopper-gpu-dpx-instructions/ |title=Boosting Dynamic Programming Performance Using NVIDIA Hopper GPU DPX Instructions |date=December 8, 2022 |last1=Tirumala |first1=Ajay |last2=Eaton |first2=Joe |last3=Tyrlik |first3=Matt |publisher=[[Nvidia]] |access-date=May 29, 2023}}</ref> Similarly, Hopper speeds up implementations of the [[Needleman–Wunsch algorithm]].<ref>{{Cite web |url=https://blogs.nvidia.com/blog/2022/03/22/nvidia-hopper-accelerates-dynamic-programming-using-dpx-instructions/ |title=NVIDIA Hopper GPU Architecture Accelerates Dynamic Programming Up to 40x Using New DPX Instructions |date=March 22, 2022 |last=Harris |first=Dion |publisher=[[Nvidia]] |access-date=May 29, 2023}}</ref>

===Transformer engine===
The Hopper architecture was the first Nvidia architecture to implement the transformer engine.<ref name=":0">{{Cite web |last=Salvator |first=Dave |date=March 22, 2022 |title=H100 Transformer Engine Supercharges AI Training, Delivering Up to 6x Higher Performance Without Losing Accuracy |url=https://blogs.nvidia.com/blog/2022/03/22/h100-transformer-engine/ |access-date=May 29, 2023 |publisher=[[Nvidia]]}}</ref> The transformer engine accelerates computations by dynamically reducing them from higher numerical precisions (i.e., FP16) to lower precisions that are faster to perform (i.e., FP8) when the loss in precision is deemed acceptable.<ref name=":0" /> The transformer engine is also capable of dynamically allocating bits in the chosen precision to either the mantissa or exponent at runtime to maximize precision.<ref name=":1">{{Cite web |title=Nvidia’s Next GPU Shows That Transformers Are Transforming AI - IEEE Spectrum |url=https://spectrum.ieee.org/nvidias-next-gpu-shows-that-transformers-are-transforming-ai |access-date=2024-10-23 |website=spectrum.ieee.org |language=en}}</ref>

===Power efficiency===
The SXM5 form factor H100 has a [[thermal design power]] (TDP) of 700 [[watt]]s. With regards to its asynchrony, the Hopper architecture may attain high degrees of utilization and thus may have a better performance-per-watt.{{Sfn|Elster|Haugdahl|2022|p=8}}

==Grace Hopper==
{{Infobox GPU microarchitecture
|name          = Grace Hopper GH200
|codename      = Grace Hopper
|created       = 2023
<!--|transistors1  = approx. 200B total<ref>{{Cite web |url=https://www.tomshardware.com/news/nvidia-unveils-dgx-gh200-supercomputer-and-mgx-systems-grace-hopper-superchips-in-production |title=Nvidia Unveils DGX GH200 Supercomputer, Grace Hopper Superchips in Production |date=2023-05-29 |website=[[Tom's Hardware]]}}</ref>-->
|compute      = GPU: 132 Hopper SMs<br/>CPU: 72 [[ARM Neoverse|Neoverse V2]] cores
|shader-clock = 1980 MHz
|designfirm             = [[Nvidia]]
|manuf1                 = [[TSMC]]
|process                = TSMC [[5 nm process|4N]]
|memory-support         = GPU: 96&nbsp;GB HBM3 or 144&nbsp;GB HBM3e<br/>CPU: 480&nbsp;GB LPDDR5X
}}
The GH200 combines a Hopper-based H100 GPU with a Grace-based 72-core CPU on a single module. The total power draw of the module is up to 1000 W. CPU and GPU are connected via NVLink, which provides memory coherence between CPU and GPU memory.<ref>{{Cite web |url=https://www.anandtech.com/show/18877/nvidia-grace-hopper-has-entered-full-production-announcing-dgx-gh200-ai-supercomputer |title=NVIDIA: Grace Hopper Has Entered Full Production & Announcing DGX GH200 AI Supercomputer |date=2023-05-29 |website=Anandtech}}</ref>

==History==
In November 2019, a well-known [[Twitter]] account posted a tweet revealing that the next architecture after [[Ampere (microarchitecture)|Ampere]] would be called Hopper, named after computer scientist and [[United States Navy]] rear admiral [[Grace Hopper]], one of the first programmers of the [[Harvard Mark I]]. The account stated that Hopper would be based on a [[multi-chip module]] design, which would result in a yield gain with lower wastage.<ref>{{Cite web |url=https://wccftech.com/nvidia-hopper-gpu-mcm-leaked/ |title=NVIDIA Next Generation Hopper GPU Leaked – Based On MCM Design, Launching After Ampere |date=November 16, 2019 |last=Pirzada |first=Usman |work=Wccftech |access-date=May 29, 2023}}</ref>

During the 2022 [[Nvidia GTC]], Nvidia officially announced Hopper.<ref>{{Cite web |url=https://www.theverge.com/2022/3/22/22989182/nvidia-ai-hopper-architecture-h100-gpu-eos-supercomputer |title=Nvidia reveals H100 GPU for AI and teases 'world's fastest AI supercomputer' |date=March 22, 2022 |last=Vincent |first=James |work=[[The Verge]] |access-date=May 29, 2023}}</ref>

In late 2022, Due to [[United States New Export Controls on Advanced Computing and Semiconductors to China|US regulations]] that limited the export of chips to the [[People's Republic of China]], adapted the H100 chip to the Chinese market with the H800. This model has lower bandwidth compared to the original H100 model.<ref>{{Cite news |title=Nvidia tweaks flagship H100 chip for export to China as H800 |url=https://www.reuters.com/technology/nvidia-tweaks-flagship-h100-chip-export-china-h800-2023-03-21/ |archive-url=http://web.archive.org/web/20231122061905/https://www.reuters.com/technology/nvidia-tweaks-flagship-h100-chip-export-china-h800-2023-03-21/ |archive-date=2023-11-22 |access-date=2025-01-28 |work=Reuters |language=en-US}}</ref><ref>{{Cite news |title=NVIDIA Prepares H800 Adaptation of H100 GPU for the Chinese Market |url=https://www.techpowerup.com/306390/nvidia-prepares-h800-adaptation-of-h100-gpu-for-the-chinese-market |archive-url=http://web.archive.org/web/20230902072201/https://www.techpowerup.com/306390/nvidia-prepares-h800-adaptation-of-h100-gpu-for-the-chinese-market |archive-date=2023-09-02 |access-date=2025-01-28 |work=TechPowerUp |language=en}}</ref> In late 2023, the US government announced new restrictions on the export of AI chips to China, including the [[Ampere (microarchitecture)|A800]] and H800 models.<ref>{{Cite web |last=Leswing |first=Kif |date=2023-10-17 |title=U.S. curbs export of more AI chips, including Nvidia H800, to China |url=https://www.cnbc.com/2023/10/17/us-bans-export-of-more-ai-chips-including-nvidia-h800-to-china.html |access-date=2025-01-28 |website=CNBC |language=en}}</ref>

By 2023, during the [[AI boom]], H100s were in great demand. [[Larry Ellison]] of [[Oracle Corporation]] said that year that at a dinner with Nvidia CEO [[Jensen Huang]], he and [[Elon Musk]] of [[Tesla, Inc.]] and [[xAI (company)|xAI]] "were begging" for H100s, "I guess is the best way to describe it. An hour of sushi and begging".<ref name="fitch20240226">{{Cite news |last=Fitch |first=Asa |date=2024-02-26 |title=Nvidia's Stunning Ascent Has Also Made It a Giant Target |url=https://www.wsj.com/tech/ai/nvidia-ceo-jensen-huang-vision-company-f05db212 |access-date=2024-02-27 |work=The Wall Street Journal |language=en-US}}</ref>

In January 2024, [[Raymond James Financial]] analysts estimated that Nvidia was selling the H100 GPU in the price range of $25,000 to $30,000 each, while on eBay, individual H100s cost over $40,000.<ref name="Vanian">{{cite news |last1=Vanian |first1=Jonathan |title=Mark Zuckerberg indicates Meta is spending billions of dollars on Nvidia AI chips |url=https://www.cnbc.com/2024/01/18/mark-zuckerberg-indicates-meta-is-spending-billions-on-nvidia-ai-chips.html |access-date=June 6, 2024 |work=CNBC |date=January 18, 2024}}</ref> As of February 2024, Nvidia was reportedly shipping H100 GPUs to data centers in armored cars.<ref name="Bousquette">{{cite news |last1=Bousquette |first1=Isabelle |last2=Lin |first2=Belle |title=Armored Cars and Trillion Dollar Price Tags: How Some Tech Leaders Want to Solve the Chip Shortage |url=https://www.wsj.com/articles/armored-cars-and-trillion-dollar-price-tags-how-some-tech-leaders-want-to-solve-the-chip-shortage-d7c75039 |access-date=May 30, 2024 |work=The Wall Street Journal |date=February 14, 2024 |url-access=subscription}}</ref>

==H100 accelerator and DGX H100==
{{NvidiaDgxAccelerators}}

==References==
===Citations===
{{Reflist}}

===Works cited===
*{{Cite journal |last1=Elster |first1=Anne |last2=Haugdahl |first2=Tor |date=March 2022 |title=Nvidia Hopper GPU and Grace CPU Highlights |url=https://www.computer.org/csdl/magazine/cs/2022/02/09789536/1E0N2woOifC |journal=[[Computing in Science & Engineering]] |volume=24 |issue=2 |pages=95–100 |doi=10.1109/MCSE.2022.3163817 |bibcode=2022CSE....24b..95E |s2cid=249474974 |access-date=May 29, 2023|hdl=11250/3051840 |hdl-access=free }}
*{{Cite journal |last1=Fujita |first1=Kohei |last2=Yamaguchi |first2=Takuma |last3=Kikuchi |first3=Yuma |last4=Ichimura |first4=Tsuyoshi |last5=Hori |first5=Muneo |last6=Maddegedara |first6=Lalith |date=April 2023 |title=Calculation of cross-correlation function accelerated by TensorFloat-32 Tensor Core operations on NVIDIA's Ampere and Hopper GPUs |journal=Journal of Computational Science |volume=68 |doi=10.1016/j.jocs.2023.101986 |doi-access=free }}
*{{Cite book |date=April 17, 2023 |title=CUDA C++ Programming Guide |url=https://docs.nvidia.com/cuda/pdf/CUDA_C_Programming_Guide.pdf |publisher=[[Nvidia]] |ref={{Harvid|Nvidia|2023a}}}}
*{{Cite book |date=April 13, 2023 |title=Hopper Tuning Guide |url=https://docs.nvidia.com/cuda/pdf/Hopper_Tuning_Guide.pdf |publisher=[[Nvidia]] |ref={{Harvid|Nvidia|2023b}}}}
*{{Cite book |date=2022 |title=NVIDIA H100 Tensor Core GPU Architecture |url=https://nvdam.widen.net/content/tdwwiwotwr/original/gtc22-whitepaper-hopper.pdf |publisher=[[Nvidia]] |ref={{Harvid|Nvidia|2023c}} }}{{Dead link|date=August 2024 |bot=InternetArchiveBot |fix-attempted=yes }}

==Further reading==
*{{Cite journal |last=Choquette |first=Jack |date=May 2023 |title=NVIDIA Hopper H100 GPU: Scaling Performance |url=https://www.computer.org/csdl/magazine/mi/2023/03/10070122/1LvvYVP9o1q |journal=[[IEEE Micro]] |volume=43 |issue=3 |pages=9–17 |doi=10.1109/MM.2023.3256796 |s2cid=257544490 |access-date=May 29, 2023}}
*{{Cite web |url=https://spectrum.ieee.org/nvidias-next-gpu-shows-that-transformers-are-transforming-ai |title=Nvidia's Next GPU Shows That Transformers Are Transforming AI |date=April 8, 2022 |last=Moore |first=Samuel |work=[[IEEE Spectrum]] |access-date=May 29, 2023}}
*{{Cite web |url=https://www.nextplatform.com/2022/03/31/deep-dive-into-nvidias-hopper-gpu-architecture/ |title=Deep Dive Into Nvidia's "Hopper" GPU Architecture |date=March 31, 2022 |last=Morgan |first=Timothy |work=The Next Platform |access-date=May 29, 2023}}

{{Nvidia}}

[[Category:Nvidia microarchitectures]]
[[Category:Graphics microarchitectures|Nvidia Hopper]]