Prepare for your exams
Get points
Guidelines and tips
Sell on Docsity
Docsity AI

Prepare for your exams

Study with the several resources on Docsity

Earn points to download

Earn points by helping other students or get them with a premium plan

Guidelines and tips

Sell on Docsity

Docsity AI

Log in Sign up

Prepare for your exams

Study with the several resources on Docsity

Find documents

Prepare for your exams with the study notes shared by other students like you on Docsity

Search for your university

Find the specific documents for your university's exams

Docsity AINEW

Summarize your documents, ask them questions, convert them into quizzes and concept maps

Explore questions

Clear up your doubts by reading the answers to questions asked by your fellow students

Earn points to download

Earn points by helping other students or get them with a premium plan

Share documents

20 Points

For each uploaded document

Answer questions

5 Points

For each given answer (max 1 per day)

All the ways to get free points

Get points immediately

Choose a premium plan with all the points you need

Study Opportunities

Choose your next study program

Get in touch with the best universities in the world. Search through thousands of universities and official partners

Community

Ask the community

Ask the community for help and clear up your study doubts

Free resources

Our save-the-student-ebooks!

Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors

Exploring Condor: A Parallel Computing System for Harvesting Idle Cycles - Prof. Jeffrey K, Study notes of Computer Science

University of Maryland Computer Science

Prof. Jeffrey K. Hollingsworth

An overview of condor, a parallel computing system designed to exploit idle cycles on workstations and dedicated clusters. The system manages both resources and resource requests, using mechanisms such as classad matchmaking, process checkpoint/restart, and remote system calls. Issues related to scheduling, transparency, and checkpoints, as well as condor's standard universe and process checkpointing.

Typology: Study notes

Pre 2010

Uploaded on 02/13/2009

koofers-user-5t0 🇺🇸

10 documents

1 / 42

This page cannot be seen from the preview

Don't miss anything!

1

Computing Environment

zCost Effective High Performance Computing

– Dedicated servers are expensive

– Non-dedicated machines are useful

• high processing power(~1GHz), fast network (100Mbps+)

• Long idle time(~50%), low resource usage

Machines in office

Need cycles to run

my simulations

Computer Lab

Supercomputer

Clustered server

W/S’s and PC’s

Network

Discover Study notes of Computer Science University of Maryland

Partial preview of the text

Download Exploring Condor: A Parallel Computing System for Harvesting Idle Cycles - Prof. Jeffrey K and more Study notes Computer Science in PDF only on Docsity!

Computing Environment

z^

Cost Effective High Performance Computing–

Dedicated servers are expensive– Non-dedicated machines are useful

-^ high processing power(~1GHz), fast network (100Mbps+) -^ Long idle time(~50%), low resource usage

Machines in office

Need cycles to runmy simulations

Computer Lab

Supercomputer

Clustered server

W/S’s and PC’s

Network

OS Support For Parallel Computing

z^

Many applications need raw compute power– Computer H/W and S/W Simulations– Scientific/Engineering Computation– Data Mining, Optimization problems

z^

Goal– Exploit computation cycles on idle workstations

z^

Projects– Condor– Linger-Longer

What Is Condor?

z^

Condor– Exploits computation cycles in collections of

workstations• dedicated clusters
- Manages both
  - resources (machines)• resource requests (jobs)
    - Has several mechanisms
      - ClassAd Matchmaking• Process checkpoint/ restart / migration• Remote System Calls• Grid Awareness
        
        Scalable to thousands of jobs / machines

Condor – Dedicated Resources

z^

Dedicated Resources– Compute Clusters

z^

Manage– Node monitoring,

scheduling

Job launch, monitor &

cleanup

Mechanisms in Condor

z^

Transparent Process Checkpoint / Restart

z^

Transparent Process Migration

z^

Transparent Redirection of I/O– Condor’s Remote System Calls

CondorView Usage Graph

Some Challenges

z^

Condor does whatever it takes to run your jobs, evenif some machines…– Crash (or are disconnected)– Run out of disk space– Don’t have your software installed– Are frequently needed by others– Are far away & managed by someone else

Condor’s Standard Universe

z^

Condor can support various combinations offeatures/environments– In different “Universes”

z^

Different Universes provide different functionality– Vanilla

Run any Serial Job
- Scheduler
  - Plug in a meta-scheduler
    - Standard
      - Support for transparent process checkpoint and restart

When Will Condor Checkpoint Your Job? z^

Periodically, if desired– For fault tolerance

z^

To free the machine to do a higher priority task(higher priority job, or a job from a user with higherpriority)– Preemptive-resume scheduling

z^

When you explicitly run– condor_checkpoint– condor_vacate– condor_off– condor_restart

Condor Daemon Layout

Personal Condor / Central Manager

master

collector

negotiator

schedd

startd

= Process Spawned

Access to Data in Condor

z^

Use Shared Filesystem if available

z^

No shared filesystem?–

Remote System Calls (in the Standard Universe) – Condor File Transfer Service

Can automatically send back changed files• Atomic transfer of multiple files
- Remote I/O Proxy Socket

Standard Universe Remote System Calls z^

I/O System calls trapped– Sent back to submit machine

z^

Allows Transparent Migration Across Domains– Checkpoint on machine A, restart on B

z^

No Source Code changes required

z^

Language Independent

z^

Opportunities– For Application Steering

Condor tells customer process “how” to open files
- For compression on the fly

Job

Fork

starter

shadow Home

FileSystem

I/O Library

I/O Server

I/O Proxy

Secure Remote I/O

Local System Calls

Local I/O(Chirp)

Execution Site

Submission Site

Job Submission Machine

Job Execution Site

Job

Condor-GGridManagerGASSServer

Condor-GSched uler

PersistantJob Que ue

End UserReq uests

CondorShadowProcess forJob X

Condor-GCollector

Fork

Globus Daemons

Local Site Scheduler

[See Figure 1]

Fork

CondorDaemons Job X Condor S yste m CallTrappi ng & C heckpoint

Fork Library

Resou rceIn fo rmatio n Tra nsfe r Job XRedi rectedSystem Call

Data

Exploring Condor: A Parallel Computing System for Harvesting Idle Cycles - Prof. Jeffrey K, Study notes of Computer Science

Related documents

Partial preview of the text

Download Exploring Condor: A Parallel Computing System for Harvesting Idle Cycles - Prof. Jeffrey K and more Study notes Computer Science in PDF only on Docsity!

Computing Environment

Cost Effective High Performance Computing–

OS Support For Parallel Computing

Many applications need raw compute power– Computer H/W and S/W Simulations– Scientific/Engineering Computation– Data Mining, Optimization problems

Goal– Exploit computation cycles on idle workstations

Projects– Condor– Linger-Longer

What Is Condor?

Condor– Exploits computation cycles in collections of

Condor – Dedicated Resources

Dedicated Resources– Compute Clusters

Manage– Node monitoring,

Mechanisms in Condor

Transparent Process Checkpoint / Restart

Transparent Process Migration

Transparent Redirection of I/O– Condor’s Remote System Calls

CondorView Usage Graph

Some Challenges

Condor does whatever it takes to run your jobs, evenif some machines…– Crash (or are disconnected)– Run out of disk space– Don’t have your software installed– Are frequently needed by others– Are far away & managed by someone else

Condor’s Standard Universe

Condor can support various combinations offeatures/environments– In different “Universes”

Different Universes provide different functionality– Vanilla

When Will Condor Checkpoint Your Job? z^

Periodically, if desired– For fault tolerance

To free the machine to do a higher priority task(higher priority job, or a job from a user with higherpriority)– Preemptive-resume scheduling

When you explicitly run– condor_checkpoint– condor_vacate– condor_off– condor_restart

Condor Daemon Layout

Access to Data in Condor

Use Shared Filesystem if available

No shared filesystem?–

Standard Universe Remote System Calls z^

I/O System calls trapped– Sent back to submit machine

Allows Transparent Migration Across Domains– Checkpoint on machine A, restart on B

No Source Code changes required

Language Independent

Opportunities– For Application Steering

Job

Fork

starter

shadow Home

FileSystem

I/O Library

I/O Server

I/O Proxy

Secure Remote I/O

Local System Calls

Local I/O(Chirp)

Execution Site

Submission Site

Job