Submitted Successfully!
To reward your contribution, here is a gift for you: A free trial for our video production service.
Thank you for your contribution! You can also upload a video entry or images related to this topic.
Version Summary Created by Modification Content Size Created at Operation
1 handwiki -- 1325 2022-10-31 01:50:26

Video Upload Options

Do you have a full video?

Confirm

Are you sure to Delete?
Cite
If you have any further questions, please contact Encyclopedia Editorial Office.
HandWiki. Sun-Ni Law. Encyclopedia. Available online: https://encyclopedia.pub/entry/31975 (accessed on 05 July 2024).
HandWiki. Sun-Ni Law. Encyclopedia. Available at: https://encyclopedia.pub/entry/31975. Accessed July 05, 2024.
HandWiki. "Sun-Ni Law" Encyclopedia, https://encyclopedia.pub/entry/31975 (accessed July 05, 2024).
HandWiki. (2022, October 31). Sun-Ni Law. In Encyclopedia. https://encyclopedia.pub/entry/31975
HandWiki. "Sun-Ni Law." Encyclopedia. Web. 31 October, 2022.
Sun-Ni Law
Edit

Sun-Ni's Law (or Sun and Ni's Law, also known as memory-bounded speedup), is a memory-bounded speedup model which states that as computing power increases the corresponding increase in problem size is constrained by the system’s memory capacity. In general, as a system grows in computational power, the problems run on the system increase in size. Analogous to Amdahl's law, which says that the problem size remains constant as system sizes grow, and Gustafson's law, which proposes that the problem size should scale but be bound by a fixed amount of time, Sun-Ni's Law states the problem size should scale but be bound by the memory capacity of the system. Sun-Ni's Law was initially proposed by Xian-He Sun and Lionel Ni at the Proceedings of IEEE Supercomputing Conference 1990. With the increasing disparity between CPU speed and memory data access latency, application execution time often depends on the memory speed of the system. As predicted by Sun and Ni, data access has become the premier performance bottleneck for high-end computing. From this fact one can see the intuition behind Sun-Ni's Law, as system resources increase, applications are often bottlenecked by memory speed and bandwidth, thus an application can achieve a larger speedup by utilizing all the memory capacity in the system. Sun-Ni's Law can be applied to different layers of a memory hierarchy system, from L1 cache to main memory. Through its memory-bounded function,W=G(M), it reveals the trade-off between computing and memory in algorithm and system architecture design. All three speedup models, Sun-Ni, Gustafson, and Amdahl, provide a metric to analyze speedup for Parallel computing. Amdahl’s law focuses on the time reduction for a given fixed-size problem. Amdahl’s law states that the sequential portion of the problem (algorithm) limits the total speedup that can be achieved as system resources increase. Gustafson’s law suggests that it is beneficial to build a large-scale parallel system as the speedup can grow linearly with the system size if the problem size is scaled up to maintain a fixed execution time. Yet as memory access latency often becomes the dominant factor in an application’s execution time, applications may not scale up to meet the time bound constraint. Sun-Ni's Law, instead of constraining the problem size by time, constrains the problem by the memory capacity of the system, or in other words bounds based on memory. Sun-Ni's Law is a generalization of Amdahl's Law and Gustafson's Law. When the memory-bounded function G(M)=1, it resolves to Amdahl's law, when the memory-bounded function G(M)=m,the number of processors, it resolves to Gustafson's Law.

linearly parallel system algorithm

1. Derivation of Sun-Ni's Law

Let [math]\displaystyle{ \textstyle W^{*} }[/math] be the scaled workload under a memory space constraint. The memory bounded speedup can be defined as:

Sequential Time to Solve W*/Parallel Time to Solve W* 

Suppose [math]\displaystyle{ \textstyle f }[/math] is the portion of the workload that can be parallelized and [math]\displaystyle{ \textstyle(1-f) }[/math] is the sequential portion of the workload.

Let [math]\displaystyle{ \textstyle y = g(x) }[/math] be the function that reflects the parallel workload increase factor as the memory capacity increases m times.

Let: [math]\displaystyle{ \textstyle W = g(M) }[/math] and: [math]\displaystyle{ \textstyle W^{*} = g(m\cdot M) }[/math] where [math]\displaystyle{ \textstyle M }[/math] is the memory capacity of one node.

Thus, [math]\displaystyle{ W^{*} = g(m\cdot g^{-1}(W)) }[/math]

The memory bounded speedup is then:

[math]\displaystyle{ \frac{(1-f)W + f\cdot g(m\cdot g^{-1}(W))}{(1-f)W + \frac{f\cdot g(m\cdot g^{-1}(W))}{m}} }[/math]

For any power function [math]\displaystyle{ \textstyle g(x) = ax^{b} }[/math] and for any rational numbers a and b, we have:

[math]\displaystyle{ g(mx) = a(mx)^{b} = m^{b}\cdot ax^{b} = m^{b}g(x) = \bar{g}(m)g(x) }[/math]

where [math]\displaystyle{ \textstyle \bar{g}(m) }[/math] is the power function with the coefficient as 1.

Thus by taking the highest degree term to determine the complexity of the algorithm, we can rewrite memory bounded speedup as:

[math]\displaystyle{ \frac{(1-f)W + f\cdot \bar{g}(m)W}{(1-f)W + \frac{f\cdot \bar{g}(m)W}{m}} = \frac{(1-f) + f\cdot \bar{g}(m)}{(1-f) + \frac{f\cdot \bar{g}(m)}{m}} }[/math]

In this equation, [math]\displaystyle{ \textstyle \bar{g}(m) }[/math] represents the influence of memory change on the change in problem size.

Suppose [math]\displaystyle{ \textstyle \bar{g}(m) =1 }[/math], Then the memory-bounded speedup model reduces to Amdahl's law, since problem size is fixed or independent of resource increase.

Suppose [math]\displaystyle{ \textstyle \bar{g}(m)=m }[/math], Then the memory-bounded speedup model reduces to Gustafson's law, which means when memory capacity increases m times and the workload also increases m times all the data needed is local to every node in the system.

Often, for simplicity and for matching the notation of Amdahl's Law and Gustafson's Law, the letter G is used to represent the memory bound function [math]\displaystyle{ \textstyle \bar{g}(m) }[/math], and n replaces m. Using this notation we get:

[math]\displaystyle{ Speedup_\text{memory-bounded} = \frac{(1-f) + f\cdot G(n)}{(1-f) + \frac{f\cdot G(n)}{n}} }[/math]

2. Examples

Suppose one would like to determine the memory-bounded speedup of matrix multiplication. The memory requirement of matrix multiplication is roughly [math]\displaystyle{ \textstyle x = 3N^{2} }[/math] where N is the dimension of the two N X N source matrices. And the computation requirement is [math]\displaystyle{ 2N^{3} }[/math]

Thus we have:

[math]\displaystyle{ g(x) = 2(x/3)^{3/2} = \frac{2}{3^{3/2}}x^{3/2} }[/math] and [math]\displaystyle{ \bar{g}(x) = x^{3/2} }[/math]

Thus the memory-bounded speedup is for matrix multiplication is:

[math]\displaystyle{ \frac{(1-f) + f\cdot \bar{g}(m)}{(1-f) + \frac{f\cdot \bar{g}(m)}{m}} = \frac{(1-f) + f\cdot m^{3/2}}{(1-f) + f\cdot m^{1/2}} }[/math]

The following is another matrix multiplication example which illustrates the rapid increase in parallel execution time.[1] The execution time of a N X N matrix for a uniprocessor is:[math]\displaystyle{ O(n^{3}) }[/math]. While the memory usage is: [math]\displaystyle{ O(n^{2}) }[/math]

Suppose a 10000-by-10000 matrix takes 800 MB of memory and can be factorized in 1 hour on a uniprocessor. Now for the scaled workload suppose is possible to factorize a 320,000-by-320,000 matrix in 32 hours. The time increase is quite large, but the increase in problem size may be more valuable for someones whose premier goal is accuracy. For example, an astrophysicist may be more interested in simulating an N-body problem with as the number of particles as large as possible.[1] This example shows for computation intensive applications, memory capacity does not need to proportionally scale up with computing power.

3. Applications/Effects of Sun-Ni's Law

The memory-bounded speedup model is the first work to reveal that memory is the performance constraint for high-end computing and presents a quantitative mathematical formulation for the trade-off between memory and computing. It is based on the memory-bounded function,W=G(n), where W is the work and thus also the computation for most applications. M is the memory requirement in terms of capacity, and G is the reuse rate. W=G(M) gives a very simple, but effective, description of the relation between computation and memory requirement. From an architecture viewpoint, the memory-bounded model suggests the size, as well as speed, of the cache(s) should match the CPU performance. Today, modern microprocessors such as the Pentium Pro, Alpha 21164, Strong Arm SA110, and Longson-3A use 80% or more of their transistors for the on-chip cache rather than computing components. From an algorithm design viewpoint, we should reduce the number of memory accesses. That is, reuse the data when it is possible. The function G() gives the reuse rate. Today, the term memory bound functions has become a general term which refers to functions which involve extensive memory access.[2] Memory complexity analysis has become a discipline of computer algorithm analysis.

References

  1. David Culler; Jaswinder Pal Singh; Anoop Gupta. Parallel Computer Architecture a Hardware/Software Approach. pp. 206–207. 
  2. U.Meyer, P.Sanders, and J.F. Sibeyn (Eds.), Algorithms for Memory Hierarchies, Vol. 2625 of LNCS Springer, 2003.
More
Information
Subjects: Others
Contributor MDPI registered users' name will be linked to their SciProfiles pages. To register with us, please refer to https://encyclopedia.pub/register :
View Times: 512
Entry Collection: HandWiki
Revision: 1 time (View History)
Update Date: 31 Oct 2022
1000/1000
Video Production Service