Distributed Processing
2007-12-25 8:46:00
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Thank you everyone who helped out on this one. It would appear that there is
a considerable amount of software which exists to allow me to do what I want.
I have not been able to look at all of the suggested possibilities yet but,
based on the distribution of replies which arrived, we will probably take
a close look at 'condor' and 'pvm' from the PD world.
One company, Myrias (Alberta, Canada) also supplied me with documentation
within a few days about their product which looks quite promising.
In total, I received 30 answers. I have listed them below as a summary
and then appended the full responses at the end for those who may be
interested. The number following the product indicates the mentions given
in the repsonses.
Public Domain
-------------
Condor (6) Extra library to be compiled in. Migrates jobs to a number
of available machines. Will not use machines whose load
exceeds some preset limit.
Net Linda (3) Provides 6 calls to add net parallelism to C programs.
PVM (7) Allows a collection of heterogeneous computers to be used
as a coherent and flexible concurrent computational resource.
Programs written in C, C++ or Fortran access PVM through
library routines.
Parallaxis (1)
lb (1)
ISIS (1)
DQS (2)
Hence (1)
Express (3)
Commercial
----------
ConnectQ (1)
LSF (1)
Load Balancer (4)
Netmake (3)
NetShare (1)
NQS (1)
PAMS(Myrias) (1)
There is apparently also an article in the April 1994 edition of Open
Computing on this topic (guess who has not received one tihs month ?)
Responses came from the following....thank you all
@nsfnet-relay.ac.uk:hasley@andy.bgsu.edu
@nsfnet-relay.ac.uk:jdd@db.toronto.edu
cecilp@vancouver.cantel.rogers.com
gautam@salvador.speech.lsu.edu
@nsfnet-relay.ac.uk:tate_j@deboom.portal.com
@kakwa.ucs.ualberta.ca:myrias.ab.ca!wtk@myrias
Thanks Again
Dave
***************************************************************************
Dr. Dave Checketts | JANET: d.g.checketts@uk.ac.bham
Computer Officer | INTERNET: checkedg@eee.bham.ac.uk
School of Elec. & Elec. Eng., |
University of Birmingham | Telephone: 021 414 4322
Birmingham, B15 2TT, | Fax: 021 414 4291
England
***************************************************************************
************************************************************************
COMPLETE LISTS OF RESPONSES FOLLOW.
*************************************
>From tommy@big.att.com Fri Apr 22 13:51:32 1994
Look on the net for a package called condor. You compile your program
with the condor library. When you run your program, it migrates across
your list of available machines. When the load average of the machine
reaches a certain level -- I think 0.13 -- it assumes that the owner
wants to use it and the process backs off and moves away. Owners of
workstations, therefore, have no reason to object to other people
running programs on their workstations this way. You can do this in
the day or night for obvious reasons.
>From pjw@ccci.com Fri Apr 22 14:12:18 1994
Shell scripts are the first simple way you can parallel process
on idle Suns. Use cmd line args to drive multiple instances
of a program, NFS to collect results in some file system.
Net Linda (offshoot of Yale research) is a set of 6 calls
that add net parallelism to C programs, provided you don't
need to share a huge amount of data. Sample ray tracing
etc code is available. Fairly cheap to universities.
----------------------------------------------------------------------
Dr. Peter J. Welcher EMAIL: pjw@ccci.com
Chesapeake Computer Consultants, Inc. PHONE: (410) 266-5686
2816 Southaven Drive or: (410) 573-1751
Annapolis, MD 21401 FAX: (410) 573-1751
----------------------------------------------------------------------
>From @nsfnet-relay.ac.uk:hasley@andy.bgsu.edu Fri Apr 22 14:16:04 1994
There are a number of programs out there that can handle such
work. I looked at Condor from the U. of Wisconsin a while back,
but it got pushed away by official work before I could finish
the installation. It looked like a nice package, but I can't
give an adequate review.
(And Archie isn't responding so I can't tell you where it is...
poke, poke ... Try 'ftp.cs.wisc.edu', directory "condor".)
John Hasley
>From ems@ccrl.nj.nec.com Fri Apr 22 14:30:05 1994
One commercial package we looked into recently here is ConnectQ from
Sterling Software. It support multiple architectures. We heard of it through
Silicon Graphics. Contact Mark Maxwell (PST) at 800-455-9273 or 415-390-3523
Ed Strong
>From @nsfnet-relay.ac.uk:jdd@db.toronto.edu Fri Apr 22 20:05:42 1994
Call Platform Computing at 416-978-0458, or call your local DEC or Convex
salesperson. Ask for LSF.
Regards,
John
--
John DiMarco jdd@cdf.toronto.edu
Computing Disciplines Facility Systems Manager jdd@cdf.utoronto.ca
University of Toronto EA201B,(416)978-1928
***************************************************************************"
>From cecilp@vancouver.cantel.rogers.com Sat Apr 23 16:44:02 1994
There is a product called
Load Balancer
by Unison Tymlabs.
in Texas U.S.A.
Tel: (512) 478-0611
Fax: (512) 479-0735
I am not their salesman and have no relationship with them.
Regards
Cecil
>From peter.allan@aea.orgn.uk Sat Apr 23 19:16:12 1994
There is such a thing but I don't know what it is.
(The guy who half-told me about it was being unhelpful.)
Please post the summary to me, when solved.
Thanks.
--
________
___________________________________ /\ | ______| /\
| | / \ | | / \
| Peter Allan | / /\ \ | |______ / /\ \
| Email : peter.allan@aea.orgn.uk | / /__\ \ | ______| / /__\ \
| Phone : (44) 925 252684 | / ______ \ | | / ______ \
| Fax : (44) 925 252390 | / / \ \ | |______/ / \ \
|___________________________________|/_/ \_\|_________/ \_\
T E C H N O L O G Y
>From doug@perry.berkeley.edu Sat Apr 23 20:27:00 1994
Check out PVM (Parallel Virtual Machine), available from:
netlib2.cs.utk.edu
- Doug Neuhauser, doug@perry.berkeley.edu, 510-642-0931
>From poffen@San-Jose.ate.slb.com Sat Apr 23 23:08:03 1994
Depends on what the job entails. If it is "makes" of software, where there
are multiple modules to compile in a single directory, a product called
"netmake" can spawn the compiles on remote machines to do parallel builds.
The company name is called "Aggregate Computing". I don't have any other
details offhand.
Russ Poffenberger DOMAIN: poffen@San-Jose.ate.slb.com
Schlumberger Technologies ATE UUCP: {uunet,decwrl,amdahl}!sjsca4!poffen
1601 Technology Drive CIS: 72401,276
San Jose, Ca. 95110 Voice: (408)437-5254 FAX: (408)437-5246
>From gta!paul@uunet.UU.NET Sun Apr 24 06:24:09 1994
Dave,
In regards to your question about distributed processing on a SPARC there are
two solutions I know of:
1. PVM - A public domain system developed at Oak Ridge National Labs
2. Express from Parasoft Corp.
We will be using Express on a system we are currently configuring. Please
contact Arthur Hicken at Parasoft Corp. for more information. They
have an ftp server at parasoft.com that has many docuements about the
Express product. If you contact Arthur, please tell him that I sent you.
Arthur Hicken
Parasoft Corporation
818-792-9941
ahicken@parasoft.com
Paul
-----
Paul Emerson | Global Technology Associates, Inc.
President | 7198 Harbor Heights Circle
Email: paul@gta.com | Orlando, FL 32835
CIS: 72355,171 | Tel 407-296-3636 FAX 407-295-1954
<too much to leave in this summary about Express so I took the liberty of deleting it>
>From bshaw@bobasun.spdc.ti.com Sun Apr 24 16:32:22 1994
Hi Dave
I've evaluated CONDOR which is *VERY* good with the only disadvantage is
that you need the .o files of your application to relink. Excellent
checkpointing capability.
I'm presently looking at taskbroker.
Also Load Balancer sounds interesting but have not played with it.
PLEASE SUMMARIZE !!!!!
Thanks
Bob
LAWYERS do it on a trial basis.
Bob Shaw Texas Instruments Inc.
13536 North Central Expressway, Mail Station 461 Dallas, Tx 75243
bshaw@spdc.ti.com
------------------------------------
>From sommer@vsun02.ag01.Kodak.com Mon Apr 25 07:10:48 1994
Dave,
I would think you even would need a special language to do that.
Using this language, your program must be structured so that
it can be divided into different threads where each thread runs
on one machine. I've heard from a program calculating fractals
on different workstations.
language: Parallaxis
package: parallaxis
version: 2.0
parts: ?, simulator, x-based profiler
author: ?
how to get: ftp pub/parallaxis from ftp.informatik.uni-stuttgart.de
description: Parallaxis is a procedural programming language based
on Modula-2, but extended for data parallel (SIMD) programming.
The main approach for machine independent parallel programming
is to include a description of the virtual parallel machine
with each parallel algorithm.
ports: MP-1, CM-2, Sun-3, Sun-4, DECstation, HP 700, RS/6000
contact: ? Thomas Braunl <braunl@informatik.uni-stuttgart.de> ?
updated: 1992/10/23
This was taken from:
Return-Path: <avmech@clpd.kodak.com>
Xref: clpd.kodak.com comp.compilers:2161 comp.lang.misc:3496 comp.archives.admin:386 news.answers:7713
Newsgroups: comp.compilers,comp.lang.misc,comp.archives.admin,news.answers,comp.answers
Path: clpd.kodak.com!kodak!newsserver.pixel.kodak.com!psinntp!psinntp!uunet!world!iecc!compilers-sender
>From: David Muir Sharnoff <muir@idiom.berkeley.ca.us>
Subject: Catalog of compilers, interpreters, and other language tools [p2of3]
Followup-To: comp.archives.admin
Summary: Monthly posting of free language tools that include source code
Keywords: tools, FTP, administrivia
Sender: compilers-sender@iecc.cambridge.ma.us
Supersedes: <free2-May-93@comp.compilers>
Reply-To: muir@idiom.berkeley.ca.us
Organization: University of California, Berkeley
References: <free1-Jun-93@comp.compilers>
Date: Tue, 1 Jun 1993 11:00:35 GMT
Approved: compilers@iecc.cambridge.ma.us
Expires: Thu, 1 Jul 1993 23:59:00 GMT
If you cannot get hold of the complete list, I can send it to
you as well (compressed with gzip). Just let me know.
Regards
Tilman
----------------------------------------------------------------------------
#### ###### Tilman C. Sommer
### ######## OI-P+E, Software and Integration Group (SIG)
## ########## Kodak AG, Breitwiesen, D-73347 Muehlhausen/Gruibingen, Germany
# ### KODAK ## Mailcode: 5023, Building 213/OG KMX : 631-2986
## ########## Phone: ++49 (7335) 12-7677, Fax : ++49 (7335) 12-7766
### ######## KNET Phone/Fax: 631-7677/7766 PROFS: (974111) LOCKOVM1
#### ###### Internet: sommer@vsun02.ag01.kodak.com
----------------------------------------------------------------------------
>From bern@penthesilea.Uni-Trier.DE Mon Apr 25 12:49:55 1994
There's NQS out there, we use it to distribute big Batches of Jobs
across our Cluster, and it would support distributed Printing, too.
However, there's nothing for having the Jobs communicate.
Regards,
J. Bern
--
__/\_____________________________________________ ___________________________
/ \ \ / /\
/ J. \ EMail: bern@[TI.]Uni-Trier.DE / ham: DD0KZ X More Infos on me from / \
\Bern/ X.400: <---- temporarily disabled ----> / \ the X.500 Directory; \ /
\ / P. O. Box 1203, 54202 Trier, Germany / \ Pub Keys via finger \/
__\/___________________________________________/ EOF \_________________________
>From shandelm@jpmorgan.com Mon Apr 25 13:08:14 1994
PVM for the Sun (written by Convex?) or Aggregate Computing Inc or
Network Linda.
-- joel
>From fgreco@lehman.com Mon Apr 25 14:27:14 1994
Contact Aggregate Computing (US) for a commercial solution
or Dikran Kassabian (deke@ee.rochester.edu) for "lb" a public
domain solution. pvm or ISIS might also fit your needs.
Frank G.
>From tkevans@eplrx7.es.duPont.com Mon Apr 25 14:47:19 1994
Check out DQS, available via anonymous ftp from 'ftp.scri..fsu.edu' in
/pub/DQS.
There's a good tutorial to DQS in the April Issue of _Unix World's Open
Computing_.
>From feldt@phyast.nhn.uoknor.edu Mon Apr 25 15:33:09 1994
Dave,
Try DQS (Distributed Queueing System). ftp from:
ftp.psc.edu:pub/dqs/DQS-2.1.tar.Z
Good luck!
Andy Feldt
System Support Programmer
Department of Physics and Astronomy
The University of Oklahoma
>From jkays@msc.edu Mon Apr 25 15:40:32 1994
Dave - You should check into PVM, which is public domain. I have attached
the README from the most current release, PVM 3.3. You can get the
source from ORNL. I don't know the exact hostname, but it shouldn't
be too hard to find. Good luck!
jeff
--
Jeff Kays
Minnesota Supercomputer Center E-Mail: jkays@msc.edu
1200 Washington Avenue South Phone: (612) 337-3422
Minneapolis, Minnesota 55415 Fax: (612) 337-3400
"May fortune favor the foolish"
>From gautam@salvador.speech.lsu.edu Mon Apr 25 15:59:39 1994
Hi,
This site "cs.dal.ca" has a complete directory of info on
distributed processing. In think it is /pub/distributedProcessing....
You could also take a look at arjuna.......
I hope this helps.
gautam pardhy
gautam@salvador.speech.lsu.edu
>From @nsfnet-relay.ac.uk:tate_j@deboom.portal.com Mon Apr 25 16:20:27 1994
Talk to aggregate computing about NetShare
>From stern@sunrise.East.Sun.COM Mon Apr 25 16:38:15 1994
check out:
Unison-Tymlabs (Load Balancer) [NOTE: sold by freedman-sharp]
675 Almanor Avenue
Sunnyvale, CA 94086
408 245 3000
Aggregate Computing (NetShare/NetMake)
300 South Highway 169
Suite 400
Minneapolis, MN 55426
800 966 1666
info@aggregate.com
network Linda (C-Linda), PVM, Express, and Hence, of which the
last three are available for ftp.
load balancer and netshare are commercial products, understand
multiple os/machine types, and are pretty rugged. pvm/express/hence
are publicly available, and make your network look like a big
compute cluster (with ethernet-type latency between nodes).
--hal
>From Ian_MacPhedran@engr.usask.ca Mon Apr 25 16:39:48 1994
The "Condor" package may be what you want - it is available from
ftp.cs.wisc.edu via anonymous ftp. Yes, the application would have to be
linked against special libraries.
Check out the latest UNIX World (now called Open something-or-other) for
an article on distributed queuing systems.
Ian.
----------------------------------------------------------------------------
Ian MacPhedran, Engineering Computer Centre, University of Saskatchewan.
2B13 Engineering Building, U. of S. Campus, Saskatoon, Sask., CANADA S7N 0W0
macphed@dvinci.USask.CA (306) 966-4832 Ian_MacPhedran@engr.USask.CA
>From vasey@issi.com Mon Apr 25 18:17:06 1994
Your request sounds almost vebatim like the script from series of demos
we used to run at the MCC Experimental Systems Lab about 3 years ago,
where I was employed on a large-scale O-O parallel processing project.
Using a small network operating system that ran on top of SunOS (and
other UNIXes) we distributed large problems over several dozen work-
stations and cranked out supercomputer scale results quite handily.
I don't know what has happened to that particular project recently,
but it did have some government support, as well as Motorola, Boeing,
and several universities participating when I left. I suggest you
contact the Director, Rob Smith (rob@mcc.com) for further information.
++ Ron Vasey International Software Systems Inc. Vox: 512+338-5724
@issi.com 9430 Research #250, Austin TX 78759 Fax: 512+338-5757
>From jimm@csdc02.orl.mmc.com Mon Apr 25 19:29:55 1994
|------------------------------------------------------------------------------|
Look at the April 1994 'Open Computing' magazine. On pages 97 - 100, they
present a number of different 'batching' or 'queing' programs available both by
the public domain and also commercially.
--------------------------------------------------------------------------------
James R. Miller -- jimm@csdc02.orl.mmc.com -- System Administrator
Martin Marietta Corporation -- Information Systems -- Orlando, Florida
Voice: (407) 826-1348 -- Fax: (407) 356-8944
--------------------------------------------------------------------------------
>From @kakwa.ucs.ualberta.ca:myrias.ab.ca!wtk@myrias Mon Apr 25 20:43:17 1994
I was forwarded the attached message.
Sun and Myrias have just (a week and a half ago) unvailed a new
product (based on 10 years of R&D done by Myrias in the MPP world)
that exactly matches what you are asking about. I will have some
information mailed to you today - you should get it in a few days.
-Wayne
--------------------------------------------------------
Wayne T. Karpoff, General Manager wtk@myrias.ab.ca
Myrias Computer Technologies Inc. 8522 Davies Road
(403) 463-1337 Fax (403) 465-0130 Edmonton Alberta
--------------------------------------------------------
>From nitkin@ptdcs2.intel.com Tue Apr 26 00:29:24 1994
One commercial product I'm aware of is "Load Balancer". You might
want to look into it. The company is:
Unison-Tymlabs
675 Almanor Avenue
Sunnyvale, CA 94086
(408) 245-3000
European Headquarters
Harpenden, Herts
44 582 462424
I've not used the product, so I can't make any type of recommendation.
Good luck.
--
- Nate Itkin
- Portland Technology Development, Intel Corporation Aloha, Oregon
- E-mail: Nate-Itkin@ptdcs2.intel.com
>From fetrow@biostat.washington.edu Tue Apr 26 01:21:10 1994
Ask archie about "condor". Rather clever.
You have to recompile you code, and it has to be Fortran, and all it
will do is distribute jobs among the best (or nearbest) SINGLE workstations
and you need a common network file system but you get "dump files" for free.
(as in: Stop the job and restart it from where you stopped). This is
EXTREMELY handy. You can bring things down for maitenance and restart
it later.
>From rminnich@super.org Tue Apr 26 15:01:29 1994
Yes, condor might do the job for distributed processing at night. It's
used at some sites on hundreds to thousands of nodes, but in fact grabs
idle cycles 24 hours a day. At any given time at least 66% of the cycles
on a network are there for the taking, for periods of 10 or 20 minutes at
a time. Condor is good at this because it supports checkpoint/migration.
ron
rminnich@super.org | It's amazing how much code depends on
(301)-805-7451 or 7312 | 0xdeadbeef not being a valid virtual address.
>From zika@trinity.tamu.edu Thu Apr 28 20:53:27 1994
An excellent PD package is PVM which we use at our site. Here's the
README file that came with:
Hope this helps...
--Michael Zika
Nuclear Engineering
Texas A&M University
(zika@trinity.tamu.edu)
Comments
Got something to say?
You must be logged in to post a comment.
