@@@@@ @ @ @ @ @ @ @@@ @ @@ @@@@ @ @ @ @@@@ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @@ @@@@@ @ @ @ @ @ @ @ @ @ @ @ @ @@ @ @ @ @@@@@ @@@ @ @@@@ @ @ @ @@@@ @ @ @ @ @ @ @@@@ @@@@ @ @@ @@@@ @ @ @@@ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @@ @ @ @ @ @@@@@ @ @@@@ @ @ @ @ @ @ @ @ @ @ @@@@ @ @@@@ @@ @@ @@@ @@ ************************************************************************** * * * O O O O * * O O * * O O O OOO O O OOOO OOOO * * O O O O O O O O O O * * O O O OOOOO O O O O O * * O O O O O O O O O O * * OOO OOO OOO OO O OOOO O O * * * * juelich mpi system (juropa) * * * * Author: juergen.oehlschlaeger@kit.edu 30 Nov. 2012 * ************************************************************************** Short instruction, how to run parallel CORSIKA on the Juelich computer center with parallelization by MPI system: (1) run ./coconut to create an executable with parallelization by scripts by selecting `p - PARALLEL treatment of subshowers` and `1 - Special stack for shell script without MPI` i.e. `corsika72495Linux_QGSII_gheisha` (login to slkit001@juropa.fz-juelich.de). (2) switch to subdirectory run/ and (optional) rename the executable in a way to distinguish standard or thinning simulations, i.e. `mpi_corsika72495_stnd_QGSII_gheisha_runner`. (3) prepare corsika steering file `parallel-001987` for parallel air shower simulation containing keyword PARALLEL (also available by `acreinphc3.f`); example see appendix (A-3). (4) create new subdirectory csk001987/ and copy files `parallel-001987`, the mpi_..._runner executable, NUCNUCCS, EGSDAT6*, qgsdat-II-03, and sectnu-II-03 to the new subdirectory. (6) switch to the simulation subdirectory csk001987/ and submit script `juelich-001987.sh` to the mpi system (see appendix (A-6)) by typing msub juelich-001987.sh and the corresponding jobid on the Juelich computer system `juropa` is displayed; get the realistic starting time by typing mjobctrl -q starttime (7) run script `sumprocessing.sh` with the new run number as argument, i.e. ./sumprocessing.sh 1987 ...... sumlongifiles.sh001987 ...... sumlistnkginfo.sh001987 ...... llsubmit showanalyhc3.sh001987 all to be executed later in the subdirectory csk001987/. Output files are the particle data files of type `DAT001987-.........-......... and the files *.long, *.lst, *.cut, *.scratch*; after the end of the simulation system protocol files and `*.scratch*` files should be deleted (if any). ************************************************************************** (A-3) parallel steering file ============================ RUNNR 1987 PARALLEL 400. 400000. 1 T NSHOW 1 PRMPAR 703 EVTNR 1 SEED 3702 0 0 SEED 3703 0 0 SEED 3704 0 0 SEED 3705 0 0 SEED 3706 0 0 SEED 3707 0 0 ERANGE 7.4132E+07 7.4132E+07 THETAP 17.00 17.00 PHIP 26.43 26.43 OBSLEV 1452.e2 870.000 g/cm^2 MAGNET 19.71 -14.18 Auger MAXPRT 1 ECTMAP 1.E11 ECUTS 0.1000 0.1000 2.5e-4 2.5e-4 RADNKG 200.E2 HADFLG 0 0 0 0 0 2 ELMFLG T T QGSJET T 0 QGSSIG T MUMULT T MUADDI T STEPFC 1. LONGI T 5. T T HILOW 111.11 DIRECT ./ HOST juropa USER you EXIT ************************************************************************** (A-6) Juelich submit script =========================== #!/bin/bash -x #MSUB -l nodes=2:ppn=8 #MSUB -l walltime=0:3:00 # here 3 min. #MSUB -M poghosyan@kit.edu #MSUB -m ae #MSUB -N csk001987 ### start of jobscript #env #pwd #set #cd $PBS_O_WORKDIR #echo "workdir: $PBS_O_WORKDIR" # NSLOTS = nodes * ppn = 2 * 8 = 16 NSLOTS=16 mpiexec -np $NSLOTS mpi_corsika72495_stnd_QGSII_gheisha_runner parallel-001987 Juelich job control commands ================================== msub juelich-001987.sh submits script to the queue and prints the job id `jobid`; mjobctrl -q starttime jobid display the system calculated start time (and end time by adding the user given walltime) of the job with the id `jobid`; checkjob -v jobid display status of job with id `jobid`; showq display list of all jobs in the queue; **************************************************************************