Download Parallel and Distributed Computing: Fundamentals, Models, and Applications and more Lecture notes Computer Networks in PDF only on Docsity!
Faculty of Computer Engineering Informatics and
Communications
Department of Computer Engineering
Module: Parallel and Distributed Computing Code: HCE/HHE/HSE 311
Introduction to Parallel and Distributed Computing
The simultaneous growth in availability of big data and in the number of simultaneous users on the Internet places particular pressure on the need to carry out computing tasks “in parallel,” or simultaneously. Parallel and distributed computing occurs across many different topic areas in computer science, including algorithms, computer architecture, networks, operating systems, and software engineering. During the early 21st^ century there was explosive growth in multiprocessor design and other strategies for complex applications to run faster. Parallel and distributed computing builds on fundamental systems concepts, such as concurrency, mutual exclusion, consistency in state/memory manipulation, message-passing, and shared-memory models.
Parallel Computing: In parallel computing multiple processors performs multiple tasks assigned to them simultaneously. Memory in parallel systems can either be shared or distributed. Parallel computing provides concurrency and saves time and money.
Distributed Computing: In distributed computing we have multiple autonomous computers which seems to the user as single system. In distributed systems there is no shared memory and computers communicate
with each other through message passing. In distributed computing a single task is divided among different computers.
Difference between Parallel Computing and Distributed Computing:
Parallel computing Distributed computing
- Many operations are performed simultaneously
System components are located at different locations
- Single computer is required Uses multiple computers
- Multiple processors perform multiple operations
Multiple computers perform multiple operations
- It may have shared or distributed memory
It have only distributed memory
- Processors communicate with each other through bus
Computers communicate with each other through message passing.
- Improves the system performance Improves system scalability, fault tolerance and resource sharing capabilities
Key Differences
- Granularity: Bit-level parallelism is the finest granularity, followed by instruction- level parallelism, and then task-level parallelism.
- Dependency: Bit-level parallelism and instruction-level parallelism require minimal dependency analysis, whereas task-level parallelism requires more complex dependency analysis.
- Scalability: Task-level parallelism is more scalable than instruction-level parallelism and bit-level parallelism, as it can take advantage of multiple processing cores and distributed systems.
Why parallel computing? The whole real world runs in dynamic nature i.e. many things happen at a certain time but at different places concurrently. This data is extensively huge to manage. Real world data needs more dynamic simulation and modeling, and for achieving the same, parallel computing is the key. Parallel computing provides concurrency and saves time and money. Complex, large datasets, and their management can be organized only and only using parallel computing‟s approach. Ensures the effective utilization of the resources. The hardware is guaranteed to be used effectively whereas in serial computation only some part of hardware was used and the rest rendered idle. Also, it is impractical to implement real-time systems using serial computing.
Applications of Parallel Computing:
- Databases and Data mining.
- Real time simulation of systems.
- Science and Engineering.
- Advanced graphics, augmented reality and virtual reality.
Limitations of Parallel Computing:
- Its addresses such as communication and synchronization between multiple sub-tasks and processes is difficult to achieve.
- The algorithms must be managed in such a way that they can be handled in the parallel mechanism.
- The algorithms or program must have low coupling and high cohesion. But it‟s difficult to create such programs.
- More technically skilled and expert programmers can code a parallelism based program well.
Future of Parallel Computing: The computational graph has undergone a great transition from serial computing to parallel computing. Tech giant such as Intel has already taken a step towards parallel computing by employing multicore processors. Parallel computation will revolutionize the way computers work in the future, for the better good. With the entire world connecting to each other even more than before, Parallel Computing does a better role in helping us stay that way. With faster networks, distributed systems, and multi-processor computers, it becomes even more necessary.
to just one.
Disadvantages of Distributed Systems It is difficult to provide adequate security in distributed systems because the nodes as well as the connections need to be secured. Some messages and data can be lost in the network while moving from one node to another. The database connected to the distributed systems is quite complicated and difficult to handle as compared to a single user system. Overloading may occur in the network if all the nodes of the distributed system try to send data at once.
Distributed Computing Models
There are certain technologies utilizing the power of distributed computing creating platforms that are flexible, reliable and scalable. These technologies are listed below:
Virtualization Service-Oriented Architecture (SOA) Grid Computing Utility Computing
Virtualization
Virtualization is a technique, which allows sharing single physical instance of an application or resource among multiple organizations or tenants (customers). It does this by assigning a logical name to a physical resource and providing a pointer to that physical resource when demanded.
The Multitenant architecture offers virtual isolation among the multiple tenants. Hence, the organizations can use and customize their application as though they each have their instances running.
. Service-Oriented Architecture (SOA) Service-Oriented Architecture helps to use applications as a service for other applications regardless the type of vendor, product or technology. Therefore, it is possible to exchange the data between applications of different vendors without additional programming or making changes to services.
The cloud computing service oriented architecture is shown in the diagram below.
Utility Computing
Utility computing is based on Pay-per-Use model. It offers computational resources on demand as a metered service. Cloud computing, grid computing, and managed IT services are based on the concept of utility computing.
Examples of real-world applications
Scientific Simulations
- Climate Modeling: Parallel computing is used to simulate complex climate models, predicting weather patterns and climate change.
- Fluid Dynamics: Distributed computing is used to simulate fluid dynamics, such as ocean currents and turbulence.
- Materials Science: Parallel computing is used to simulate the behavior of materials at the molecular level.
Data Analytics
- Big Data Processing: Distributed computing is used to process large datasets, such as those generated by social media, sensors, and IoT devices.
- Machine Learning: Parallel computing is used to train machine learning models, such as neural networks and decision trees.
- Data Mining: Distributed computing is used to analyze large datasets, identifying patterns and trends.
Artificial Intelligence
- Deep Learning: Parallel computing is used to train deep learning models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs).
- Natural Language Processing: Distributed computing is used to analyze and process large amounts of text data.
- Computer Vision: Parallel computing is used to analyze and process visual data, such as images and video.
Cryptography
- Cryptanalysis: Distributed computing is used to break encryption algorithms, such as RSA and AES.
- Secure Communication Protocols: Parallel computing is used to implement secure communication protocols, such as SSL/TLS.
Weather Forecasting
- Numerical Weather Prediction: Parallel computing is used to simulate weather patterns, predicting weather forecasts.
- Climate Modeling: Distributed computing is used to simulate climate models, predicting long-term climate trends.
Finance and Economics
- Risk Analysis: Distributed computing is used to analyze financial risk, predicting potential losses and gains.
- Portfolio Optimization: Parallel computing is used to optimize investment portfolios, maximizing returns and minimizing risk.
Healthcare and Bioinformatics
- Genomics: Distributed computing is used to analyze genomic data, identifying genetic variants and predicting disease susceptibility.
- Medical Imaging: Parallel computing is used to analyze medical images, such as MRI and CT scans.
Other Applications
- Gaming: Parallel computing is used to render graphics and simulate game physics.
- Video Processing: Distributed computing is used to process video data, such as video encoding and decoding.
- Robotics: Parallel computing is used to control robots, simulating motion and predicting behaviour.
