ASTRA

A Space Charge Tracking Algorithm

- parallel edition -

Sascha Meykopff (DESY)

Since Lars left us, I'm the only parallelized ASTRA maintainer. The parallelized version is still under development. Bug reports are welcome. Send an email to SASCHA (DOT) MEYKOPFF (AT) DESY (DOT) DE for bug reports and binary download requests. Please contact Klaus Flöttmann at DESY for information on ASTRA in general. Outdated information and old parallelized ASTRA downloads are still available from the old homepage.

2011-12-01: ASTRA parallel patch run time behaviour

I've done a huge benchmark on our ferrari3 cluster. I used the PITZ test1 with 20K, 200K, 500K, 1M and 2M particles and 3 different space charge grid sizes. All these tests are done on 1, 4, 8, 16, 32, 64, 128, 192 and 256 cores. The single core test worked with the serial ASTRA version and the other used the parallel patch R57 ASTRA. (green line: Nrad=18;Nlong_in=40, blue line: Nrad=30;Nlong_in=60, red line: Nrad=40;Nlong_in=100)

emission_test_1-256_cores

2011-11-25: Updated parallel patch version (r58)

Upstream update: new WAKE namelist

2011-11-18: Public release of parallel ASTRA patch set (R57)

This is the first public release of the parallel ASTRA patch set. The serial ASTRA and the old parallel ASTRA are too different to merge all changes from Klaus into the parallel version. I created a new patch set to convert the serial version into a multi-processor version. Now it's easy to update the parallel version.

The current patch set supports nearly all parameters from the offical ASTRA manual. Currently the second emission is disabled and the debunch parameter is untested. The patch set supports parallel emission of particles. This dosen't work with the old parallel version. Sometimes the decision if the space charge field will be scaled or new calculated depends on the number of used cores. Then the number of iterations are different. Please decrease the parameter MAX_SCALE if you need to fix this. In the most cases this affects only the FFT space charge solver with long distance tracking. Please request a new version if you need rotated elements.

I ran the 'test1' benchmark from Photo Injector Test Facility at DESY, Location Zeuthen (PITZ) on the ferrari3 cluster. This test emitts 200k particles and track them 10 meters. The calculation of space charge is enabled with N_long_in=80 and Nrad=40 (Files: test1.in, Generator.in, Cavity, Cavity, Solenoid). The run time of the serial ASTRA version is 429 minutes.

PITZ test1 on ferrari3 benchmark

Click on the result times to view the run time log file. The result table includes the loading time of all input files.

Cores: 2 4 8 16 32 64 96 128 160 192 256
R421 (min) 162:11 109:34 64:02 42:21 27:18 22:26 18:14 16:50 16:28 17:31 18:49
R57 (min) 179:38 118:38 57:24 41:57 26:27 19:09 16:05 14:39 13:45 13:02 12:40
The old version is faster and all optimations are gone in the patch set. But the new patch set scales better than the old parallel version because of the parallel emission of particles and the parallel write of output distributions. If you use the FFT space charge solver expect not so nice scaling values.

2011-11-08: Serial ASTRA archive and current manual (UPDATE: 2011-11-25)

I've created an archive of my serial ASTRA linux binaries (x86-64). Please use them if you need to compare your results. I've build these binaries from source with the Intel compiler. If you need binaries for a different plattform visit the download directory of Klaus Flöttmann.

The current serial ASTRA documentation is available here.

2011-07-29: First results of new code

I have done some comparison benchmarks with my new code. This code uses the latest code from Klaus and the parallel patch from me. I tried to determine the new parallel emission behaviour of the cylindrical symmetric space charge algorithm.

20K particles 200K particles 500K particles 1M particles 2M particles
Single cpu run:
Nrad=18;Nlong=40 2:55 13:52 31:39 1:01:42 2:05:21
Nrad=30;Nlong=60 5:51 17:43 36:46 1:08:08 2:11:57
Nrad=40;Nlong=100 12:15 24:48 44:14 1:16:47 2:21:05
4 cpu run:
Nrad=18;Nlong=40 1:22 4:27 9:49 0:18:05 0:35:23
Nrad=30;Nlong=60 3:09 6:30 12:05 0:21:14 0:38:15
Nrad=40;Nlong=100 7:10 10:43 16:38 0:25:44 0:44:51
The tests were done on my desktop computer with an Intel Q9650.

emission_test

The new code is not yet finished.

2011-04-05: Updated development version (r420)

Crash fix in namelist ERROR. Thanks to Zhangmeng/SINAP, CAS for reporting. Please be careful if you use the random error code. The results still looks odd.

Updated Q_Schottky code. Thanks to Bolko Beutner/PSI for reporting.

Important notice: We are planning to merge the serial astra version from Klaus and my parallel code. This work is scheduled for summer 2011.

2010-04-28: Updated development version (r405)

I've updated the input parser. Now it's possible to use a comma separated input list.

  • Removed the one parameter/line restriction

2010-03-30: Updated development version (r404)

I fixed some bugs. Thanks to M. Dohlus/DESY and J.P. Carneiro/Fermilab for reporting.

  • Shorten cpu usage summary
  • Fixed bug concerning 3d field map cavity definitions
  • Fixed bug concerning module definitons

2010-02-26: Development version (r398) scales much better now

I've parallelized the calculation of the field coefficients for a cylindrical grid. Astra scales now much better. (Update 2010-03-02): Here is a (r399) benchmark with a file from Photo Injector Test Facility at DESY, Location Zeuthen (PITZ) (200000 particles; enabled 2D space charge; enabled mirror charge) The r386 tests were done by Igor Zagorodnov.

pitz18_test1_r399_benchmark

2010-02-03: Updated development version (r389)

I've backported the changes in the dipole calculation code from Klaus to my development version.

The dipole support is UNTESTED, EXPERIMENTAL, UNDOCUMENTED and nobody assumes responsibility for the calculation results! USE IT AT YOUR OWN RISK!

  • Dipole calculations similar to serial version
  • Input parser supports complex values

2010-01-15: Update stable version

There is a stable update available. The 2D space charge solvers of the serial and parallel versions are now the same. Currently I distribute Linux/Amd64 binaries with mpich2 or openmpi bindings only. Please send me an email for download information.

  • Fixed 2D space charge solver
  • Linux/Amd64 binaries with mpich2 or openmpi support

2010-01-15: New development version

The new development version is much faster than the stable one. My flash start-to-end simulation with 20k particles is 30% faster and the simulation with 200k particles is nearly 50% faster. I've rewritten the input parser. The configuration file is no longer strictly a fortan namelist, although it has a similar syntax. Namelist specifiers, terminators and unkown parameters will be ignored. The parameter order doesn't matter. You can mix magnet definitions and other parameters. The loop scan system has been removed. Instead you now supply a number in the first column of an input line to start a parameter scan. This is an example from a file for a scan with 3 different valuess for one of the parameters: 

      
	  Q_length(1)=0.1850
	1 Q_K(1)=2.867  ! used in the first run
	2 Q_K(1)=2.877  ! second run
	3 Q_K(1)=2.887  ! last run
	  Q_Bore(1)=0.0000001
	  Q_pos(1)=13.9752

Update: (R405):

There is a new restriction: only a single parameter per line is allowed.
If you're interested in the new version please send me an email.
  • Up to 50% faster
  • Removed fortran namelist restrictions
  • Easy to handle parameter scan
  • Linux/Amd64 binaries with mpich2 or openmpi support

2010-01-05: I'm not dead!

The parallelized astra is still alive.