Cosmology Software

Manual started by Victor Travieso, May 2004 Last updated by Andrey Kaliazin, July 18, 2011

This page lists popular scientific software installed on COSMOS. All custom-built applications and libraries are installed under COSMOLIB=/home/cosmos/share/$ARCH/$PP/ folder, so that the versions available depend on the current platform (x86-64 for universe and ia64 - for cosmos) and SGI ProPack level (PP7, as of 2011). The way to access the relevant version of the software stack, is to load the module 'cosmolib', which is done automatically at the login via the /etc/profile script, so users don't have to worry about it. Additional modules can be loaded automatically via personal .bashrc.local script:

$ module load healpix

Once the module is loaded, you should have the right environmental variables to use the programs (eg. anafast, synfast from Healpix) and link with the libraries (eg. cfitsio using the linker flag -lcfitsio ) in the repository. If you want to see the exact commands that the module performs, you can type 'module show healpix' in the command line.

To see what environments are available at the moment and which are loaded, use the corresponding commands:

$ module avail

------------------------------------------ /home/cosmos/share/x86_64/pp7/modules -------------------------------------------
SuiteSparse/3.6.1(default)                     healpix/2.15a                                  nagc.old
autoconf/2.68(default)                         healpix/2.20a                                  nagf
boost/1_47_0-gcc                               healpix/2.20a-intel12(default)                 nagf.old
boost/1_47_0-intel                             healpix/2.20a-test                             nagsmp
boost/1_49_0-intel(default)                    icomp/10.1.026                                 nagsmp.old
cfitsio/3.25                                   icomp/11.1.075                                 netcdf/3.6.2
cfitsio/3.28-intel11                           icomp/12.1.1.256                               netcdf/4.1.1(default)
cfitsio/3.29-intel12(default)                  icomp/12.1.2.273                               ompp/0.7.1(default)
cosmolib                                       icomp/12.1.3.293(default)                      papi/3.7.2(default)
cosmos.common                                  latest                                         papi/4.1.2.1
fftw2/2.1.5-intel11                            libpfm/3.10(default)                           perfsuite/1.0.0
fftw2/2.1.5-intel12(default)                   libpfm/4.1.0                                   perfsuite/1.0.0a4
fftw3/3.2.2-ompc                               nag/all(default)                               perfsuite/1.0.0a4-papi-cvs-2011-05-20(default)
fftw3/3.2.2-openmp-obsolete                    nag/c                                          pgplot/5.2.2-gnu
fftw3/3.2.2-thread(default)                    nag/c-09-acml                                  pgplot/5.2.2-intel(default)
fftw3/3.3.1-intel12                            nag/c-09-mkl                                   ruby/1.9.2-p180(default)
gsl/1.14                                       nag/f                                          shtools/2.5(default)
gsl/1.15-intel11                               nag/f-22-mkl                                   sundials/2.3.0
gsl/1.15-intel12(default)                      nag/f-23-mkl                                   sundials/2.4.0(default)
hdf5/1.6.10                                    nag/mkl                                        tau/2.19.2(default)
hdf5/1.8.5                                     nag/smp                                        tipsy/2.2.3c(default)
hdf5/1.8.6                                     nag/smp-22-acml                                vapor/1.5.2
hdf5/1.8.6-cxx                                 nag/smp-22-mkl                                 vapor/2.0.2-gnu(default)
hdf5/1.8.8                                     nag.old                                        visit/2.1.1
hdf5/1.8.8-intel12(default)                    nagc                                           visit/2.3.2(default)
---------------------------------------------- /usr/share/modules/modulefiles ----------------------------------------------
MPInside/3.5.1             module-cvs                 mpich-ch-p4mpd             perfboost                  sgi-upc-devel/1.05
chkfeature                 module-info                mpiplace/1.01              perfcatcher                torque/sysdefault(default)
dot                        modules                    mpt/2.05                   scotch/5.1.11              use.own
moab/sysdefault(default)   mpich-ch-p4                null                       sgi-upc/1.05

$ module list
Currently Loaded Modulefiles:
  1) mpt/2.05                3) icomp/12.1.3.293        5) cfitsio/3.29-intel12
  2) cosmos.common           4) cosmolib                6) healpix/2.20a-intel12

Where source distributions of the applications are available, we have tried to create robust builds optimised for the Altix through a careful selection of configuration and compiler options. Although we try to ensure that all the software is well tested and bug free, the best guarantee is having the applications widely tested throughout the consortium. Please let us know if you find any omissions or bugs.

We welcome feedback about the library, so do contact us with comments or suggestions, eg. other cosmology related software that you would like to see included in the repository, to tell us about updates that we might have missed, etc.

Numerical Libraries

High performance, parallel numerical libraries from NAG and Intel are installed on COSMOS. These are strongly recommended to make the most out of the Altix, as they have been tuned specifically for the platform. If you have need to perform some common numerical task repeatedly in your program (eg. FFTs, ODEs, Linear Algebra) it will be worth exploring the libraries and trying to use the routines included in them. An added benefit is that a good amount of the routines can execute in parallel. In addition, the general purpose GNU Scientific Library is also available.

For more information on the libraries available on COSMOS, refer to the COSMOS developer guide.

FFT Libraries

Cosmos Developer Guide

Variants: FFTW2, FFTW3, NAG(SMP), MKL, SCSL(obsolete)

To perform FFTs efficiently on the Altix, the above vendor libraries generally offer the best performance and scalability. The vendor libraries are highly optimized for COSMOS - both at the chip and architecture level - and are general enough to satisfy most FFT calculations. All libraries offer shared memory parallelism via OpenMP constructs, and the benefits of parallel processing are accessible with no extra complications - just a change in the linking flags and setting the environmental variable OMP_NUM_THREADS. A drawback of the existing FFT libraries is that no standard interface exists for FFTs, and the calling conventions of each library differ widely. However, the extra effort of adapting a program to use one of the libraries is often compensated by the considerable gains in performance. This said, we recommend using FFTW2 API, which is perhaps most widely spread, supported actively by the makers and also Intel, which provides FFTW wrappers for their own DFTI libraries (part of MKL)

The NAG SMP library FFT routines are very straightforward to use, and the interface is detailed in the library documentation, available from the the NAG website. There are several versions of the NAG libraries on COSMOS, but we recommend the NAG SMP library, which can be called from both C and Fortran, for the best performance. On COSMOS, you can access ready made code examples that make use of the FFT routines (type 'nagsmpexample_scs' followed by the routine name that you are interested in, eg. 'nagfexample c06puf' to see an example of 3D complex FFTs).

Now obsolete, being superseded by Intel MKL, the SGI SCS Library is still available (on cosmos only!) , providing easy to use parallel FFT routines. The documentation for the libraries is available online, or accesible via the man pages on cosmos (eg. 'man scsl' and

'man
intro_fft'

).

The MKL library provides a comprehensive and very efficient FFT library (also with a Fortran90 interface). For further details, refer to the MKL Documentation online.

For the compilation flags of the various vendor libraries please refer to the Developer Guide.

FFTW3

http://www.fftw.org

Current version: 3.2.2
Notes: Static and dynamic versions, single and double precision, OpenMP parallel.
Environment modules: cosmolib, fftw3 (optional)

The FFTW3 library is an open source, high performance library for real and complex Fast Fourier Transforms of any dimension. You can link to the FFTW3 library easily from Fortran, C or C++, and on COSMOS the library can perform FFTs in parallel through OpenMP. You can see examples of use from both the C and Fortran interfaces in the FFTW3 documentation.

To link against the library on COSMOS, you should compile the program using the following linker flags for the double precision version:

$ icc my_fft.c -lfftw3 -openmp

or for the single precision (float) version:

$ icc my_fft.c  -lfftw3f -openmp

The number of processors used by the FFT routines is determined programmatically and through the environmental variable OMP_NUM_THREADS.

FFTW2

http://www.fftw.org

Current version : 2.1.5
Notes: Static versions, separate real and complex libraries, single and double precision, MPI parallel.
Environment modules: cosmolib, fftw2 (optional)

The older version of FFTW is also installed on COSMOS, mainly to provide access to the MPI FFTs not yet implemented in FFTW3. However, for any other transforms, it is recommended that you use a vendor library or version 3 of the FFTW library for greater performance.

The FFTW2 library has both Fortran and C interfaces making it straightforward to use from any of these languages. For a detailed description of the library routines and code examples, refer to the online manual.

Note that the library comes in two different versions, with separate library files for real and complex transforms - librfftw.a and libfftw.a respectively. On COSMOS, the libraries and include files are also prefixed (d or s) according to the precision of the library, so in general you must use the correct include files and linking flags to select the appropiate version (-libfftw and fftw.h default to complex, double precision transforms). Eg. :

$ icc my_fft.c -ldfftw
   Links with the double precision, complex fftw library. Use #include <dfftw.h>
       
$ icc my_fft.c -ldrfftw
   Links with the double precision, real fftw library. Use #include <drfftw.h>
       
$ icc my_fft.c -lsfftw
   Links with the single precision, complex fftw library. Use #include <sfftw.h>
       
$ icc my_fft.c -lsrfftw
   Links with the single precision, real fftw library. Use #include <srfftw.h>
       
$ icc my_fft.c -ldfftw_mpi -ldfftw -lmpi
   Links with the double precision, complex, parallel fftw library. Use #include <dfftw_mpi.h>

COSMOMC

http://cosmologist.info

Current version: Jan'12
Notes: New WMAP 7yr version.
Addtional environment modules: cfitsio healpix

The latest version of CosmoMC (Jan'12) accomodates the WMAP seven year data, is optimized for hybrid MPI and OpenMP parallelism, and can run very efficiently on the Altix. You can download the latest version from the official site.

For best results, we recommend compiling the distribution with the latest Intel compiler (module: latest - should be loaded by default) and running with MPI parallelism. If you are using a modified camb, or if you want to build your own binary, the recommended build options for COSMOS are as follows:

(Edit camb/Makefile to use the following lines - build with 'make all')
       
# Intel 12.x on COSMOS machines (currently - universe, x86_64)
F90C = ifort
FFLAGS = -g -xHost -O3 -align -ansi-alias -traceback -check all -openmp -O3 -w -fpp2 -DMPI -lmpi

For camb_fits, the necessary additions are:

(Edit camb/Makefile to use the following lines)

#Settings for building camb_fits
#Location of FITSIO and name of library
FITSDIR = $(CFITSIO)
FITSLIB = cfitsio
#Location of HEALPIX for building camb_fits
HEALPIXDIR = $(HEALPIX)

For CosmoMC:

(Edit source/Makefile to use the following options)

# Use with Intel 12.x on COSMOS machines (currently - universe, x86_64)
# (It is worth checking that the default modules 'cosmolib' and others - see above - are loaded, before compiling. 
Usually they already are, set up via the system-wide .bashrc)
F90C = ifort
FFLAGS = -g -xHost -O3 -align -ansi-alias -traceback -check all -w -fpp2 -DMPI 
LAPACKL =  -mkl=sequential -lmpi

To use the WMAP 7yr data, you should build CosmoMC against the centrally installed libraries using:

(Edit source/Makefile to set up the paths as follows)

#Edit for CFITSIO, WMAP directories
cfitsio = $(CFITSIO)
WMAP = $(COSMOLIB)/WMAP7
GSLPATH = $(GSL_ROOT)

Iif you don't want to build CosmoMC with WMAP support, you can ignore the above path, or at least comment out the WMAP variable above.

Note that there is a perl script (runCosmomc) to submit CosmoMC-type of jobs on COSMOS in a straightforward way. The script takes a few command line arguments and generates and submits the appropiate job to the queues, automating the selection of the right dplace mask. You can also use runCosmomc if you are running a modification of CosmoMC which retains the same program structure - i.e. MPI chains and possibly an OpenMP core. To see the command line options that the script takes, refer to this FAQ.

Also note that CosmoMC can now checkpoint the chains past the initial burn phase if you set'checkpoint = T' in the .ini file. This feature is strongly recommended for long runs on COSMOS in order to minimize data loss in case of a failure. Checkpointing coupled with smaller run time requirements is also a good way to get jobs running earlier and help the scheduler fill the machine to maximum capacity: If you submit CosmoMC jobs with less than the maximum run time, say 2 or 4 hours, there is a good chance your job will run much earlier as the queueing system might use it to backfill resources reserved by large jobs. Dividing a long job into smaller ones using the checkpointing option will in general reduce the time your jobs are waiting to run and give you a much better turnaround.

WMAP likelihood

http://lambda.gsfc.nasa.gov/product/map/current/

Current version: v4 (7yr)
Location: /home/cosmos/share/common/WMAP7/data
${COSMOLIB}/WMAP7 -- (library, MOD files and data)
Environment modules: cosmolib cfitsio

The WMAP likelihood software (V4, WMAP7 - became avaialble Jan'2010) used to calculate likelihoods of various models is now installed in the cosmolib. You can use the compiled version to build against other applications (eg. CosmoMC). If you are planning on using some other WMAP related software or need any of the other data files available on COSMOS, please let us know and we will install those in a central location according to demand. This will minimize the impact that multiple downloads of large datasets would have on the disk space.

This version of the software has been compiled with only the '-DOPTIMIZE' flag, and links against the latest cfitsio library.

Please note that the previous versions of the library, MOD files and data are still available from/home/cosmos/share/common/WMAP5/data and ${COSMOLIB}/WMAP5

HEALPix

http://sourceforge.net/projects/healpix/

Current version: 2.20a
Notes: Optimised build, includes the IDL tools, C interface and C++ version. OpenMP parallel.
Additional environment modules: healpix, cfitsio

The latest version of Healpix includes significant optimizations in P_lm recursion algorithm (2 to 3 times faster than version 1.22), improved IDL tools and Fortran90 visualization software (map2gif).

To use the centrally installed Healpix optimised for the Altix, you should load the healpix module in addition to the standard set (cosmolib latest) from either the command line or your startup scripts:

$ module load healpix

If you are using programs that link with the Healpix library, you will have to recompile them in order to make use of the new version. The C interface and IDL configuration variables are also included in the installation, along with the C++ version of the library and tools - after loading cosmolib, you can access the C++ build under $HEALPIX_CXX. You will need the following flags to build your programs with Healpix:

Library flags: -lhealpix -lcfitsio

To maintain compatibility with older codes, the older versions of Healpix are still available on COSMOS and can be accessed by loading a corresponding module, eg.:

$ module unload healpix
$ module load healpix/2.xxx

To see which versions available use the module avail command.

SHTOOLS

http://www.ipgp.jussieu.fr/~wieczor/SHTOOLS/SHTOOLS.html

Current version: 2.5
Notes: Built with Intel compilers. Static library only.
Environment modules: cosmolib, shtools

SHTools is a library for spherical harmonics. It can perform transforms, reconstructions, rotations of coefficients and spectral analyses on the sphere. For documentation and examples of use see the web page above. To compile against SHTools, you need the following flags:

Library flags: -lshtools -lfftw3
E.g.:
$ ifort mycode.f90 -o mycode -lshtools -lfftw3

IDL

http://www.rsinc.com/idl

Current version: 8.0
Notes: No native ia64 support. Emulation mode might be slow for large data/calculations on cosmos.
Environment modules: default

You can access IDL from COSMOS. However, note that there is no ia64 version of IDL available. The version available on COSMOS is the x86 version, working on the Itanium via the 32-bit emulation mode.Note however, that this is fairly slow.Please use instead the x86_64 nodes of COSMOS (ariel - available provisionally, since 2009).

CFITSIO

http://heasarc.gsfc.nasa.gov/docs/software/fitsio/fitsio.html

Current version: 3.29
Notes: Built with Intel Compilers, shared and static versions available.
Environment modules:cosmolib, cfitsio

Cfitsio is the standard library for writing with files in the FITS data format. The library includes both C and Fortran interfaces, and has been improved with complete support for large data files (>2.1 Gb).

To link programs using the library, you will need to use the following linker flags in your compilation:

$ icc my_fits.c -lcfitsio

The current installation of Healpix is always built against the current release of cfitsio. If you want to use the previous releases of cfitsio, eg. with an older code, you can still access it by loading correponding modules, prior to compilation/execution (older versions may vary on cosmos and universe):

$ module unload cfitsio
$ module load cfitsio/3.25

HDF5

http://hdf.ncsa.uiuc.edu/HDF5/

Current version:1.8.8
Notes: Built with Intel compilers and the MPT library for efficient parallel I/O.
Environment modules: default

HDF5 is a library for high performance data storage and manipulation. The latest version (1.8.5) has been built and optimised for the Altix using the latest Intel compilers and the MPT library for parallel I/O. The library, include files and example programs are installed under `$COSMOLIB/hdf5`. To use the library in your programs, you will need to add the following flags to your compilation:

$ icc hdf5code.c -lhdf5 -lmpi

The H5Utils, a set of programs to manipulate and convert hdf5 files from other data formats, are also installed with the library, and should be accessible once you load the module 'cosmolib'. For more information about the hdf5 file format and library, and how to use it from your Fortran or C code, refer to the HDF5 documentation.

PGPLOT

http://www.astro.caltech.edu/~tjp/pgplot/

Current version: 5.2.2
Notes: There are two builds: one(default) with ifort and another, with gcc/gfortran. See 'module avail pgplot' for details.
Environment modules: default, pgplot.

Pgplot is the standard open source graphics library for scientific applications, widely used in a variety of software packages. In order to link your application with the library, you will need to link the fortran runtime library and sometimes the X11 libraries. If you get missing symbols in the linking stage, you can use the complete flags as follows:

$ icc -g -O3 -xHost myplot.c -lpgplot -L/usr/X11R6/lib -lX11 -lpng

$ ifort -g -O3 -xHost myplot.f90 -lpgplot -L/usr/X11R6/lib -lX11 -lpng

Note, some applications require that the environmental variables PGPLOT_DIR are set - this is done automatically when the default module 'cosmolib' or specific 'pgplot' are loaded. See 'module show pgplot' for details.

Gadget-2

http://www.mpa-garching.mpg.de/gadget/

Current version : 2.0.6 - public
Location: Not installed site-wide: please download from the public location and compile
Notes:Compiles successfully with the latest Intel compilers
Environment modules: default

To build versions of Gadget-2 on COSMOS, you should edit the Makefile and add the following information:

SYSTYPE="COSMOS"

ifeq ($(SYSTYPE),"COSMOS")
CC       =   icc
OPTIMIZE = -g -O3 -xHost -align -ansi-alias -mcmodel=large
GSL_LIBS =  -lgsl -mkl=sequential
# GSL_INCL = -I$(COSMOLIB)/include
# FFTW_INCL= -I$(COSMOLIB)/include
# FFTW_LIBS= -L$(COSMOLIB)/lib
# HDF5INCL = -I$(COSMOLIB)/include
HDF5LIB  = -lhdf5 -lz
MPICHLIB = -lmpi
OPT   +=  -DDOUBLEPRECISION
OPT   +=  -DDOUBLEPRECISION_FFTW
endif

Also update the 'LIBS' variable:

LIBS   =   $(HDF5LIB) $(MPICHLIB) $(GSL_LIBS) $(FFTW_LIB)

Enzo

http://cosmos.ucsd.edu/enzo/

Current version : 1.5
Location: N/A
Notes: none
Environment modules: default

To build Enzo on Cosmos, you need to place this Make.mach.Cosmos machine config file in the 'enzo/src/' subdirectory. The latest Intel compiler is selected by default. You can build the distribution by doing:

$ ./configure
$ cd src/enzo
$ wget http://www.damtp.cam.ac.uk/cosmos/software/enzo/Make.mach.Cosmos
$ make machine-Cosmos
$ make opt-high
$ make show-config
$ make clean
$ make && make install

The Cosmos configuration is selected by the command 'make machine-Cosmos' - if the configuration is loaded successfully, the following commands should produce an output, similar to this:

$ make machine-Cosmos

 *** Execute 'gmake clean' before rebuilding executables ***

   MACHINE: COSMOS (MPI/OpenMP) platform (UK's National Cosmology Supercomputer) (Make.mach.Cosmos)

$ make opt-high io-64

 *** Execute 'gmake clean' before rebuilding executables ***

   CONFIG_OPT:                   high 
   CONFIG_IO:                    64
    
$ make show-config

   MACHINE: COSMOS platform (UK's National Cosmology Supercomputer) (Make.mach.Cosmos)

   PARAMETER_MAX_SUBGRIDS:       100000
   PARAMETER_MAX_BARYONS:        20
   PARAMETER_MAX_TASKS_PER_NODE: 8

   CONFIG_PRECISION:             64
   CONFIG_PARTICLES:             64
   CONFIG_INTEGERS:              64
   CONFIG_INITS:                 64
   CONFIG_IO:                    64
   CONFIG_USE_MPI:               yes
   CONFIG_OBJECT_MODE:           64
   CONFIG_TASKMAP:               no
   CONFIG_PACKED_AMR:            yes
   CONFIG_PACKED_MEM:            no
   CONFIG_JBPERF:                no
   CONFIG_PAPI:                  no
   CONFIG_UNIGRID_TRANSPOSE:     yes
   CONFIG_OOC_BOUNDARY:          no
   CONFIG_OPT:                   high
   CONFIG_TESTING:               no
   CONFIG_ISOBCS:                no
   CONFIG_TPVEL:                 no

$ make show-flags

CPP = /opt/intel/Compiler/11.1/046/bin/ia64/ifort
CC  = /opt/intel/Compiler/11.1/046/bin/ia64/icc
CXX = /opt/intel/Compiler/11.1/046/bin/ia64/icpc
FC  = /opt/intel/Compiler/11.1/046/bin/ia64/ifort
F90 = /opt/intel/Compiler/11.1/046/bin/ia64/ifort
LD  = /opt/intel/Compiler/11.1/046/bin/ia64/ifort

DEFINES = -DLINUX -DALTIX  -D__max_subgrids=100000 -D__max_baryons=20 -D__max_cpu_per_node=8 -DINITS64 -DLARGE_INTS -DIO_64  -DUSE_MPI   -Dp8 -Dr8  -DUSE_HDF5_GROUPS -DSINGLE_HDF5_OPEN_ON_INPUT     -DUNIGRID_TRANSPOSE

INCLUDES =  -I/home/cosmos/share-ia64.pp5/include

CPPFLAGS = -E
CFLAGS   = -openmp -O2
CXXFLAGS = -openmp -O2
FFLAGS   = -openmp -real-size 64 -integer-size 64  -O2
F90FLAGS = -openmp -real-size 64 -integer-size 64  -O2
LDFLAGS  = -openmp -nofor-main -cxxlib -O2

LIBS     = -lhdf5 -lz -lscs_mp -lcxaguard -lpthread -lmpi

With the default Cosmos machine config file mentioned above, the Enzo application is built multithreaded with OpenMP. In the current version, only FFT routines in the Enzo code are thread-enabled and it is not apparent, in which cases this additional parallelisation is beneficial or desireable. More testing is needed to figure that out.

OpenMP support can be disabled completely, by using this Make.mach.Cosmos_nomp machine config file instead. Use

$ wget http://www.damtp.cam.ac.uk/cosmos/software/enzo/Make.mach.Cosmos_nomp 
$ make machine-Cosmos_nomp

and then follow with the commands, as in the example above.

MATLAB

http://www.mathworks.com/

Current version: 2010b (on universe), 2007b (on cosmos)
Notes: No native ia64 support. Runs in 32-bit emulation mode on cosmos.

Matlab doesn't currently support the ia64 architecture, and must run in x86 emulation mode on the Itanium2. In most cases this will be ok, but if you are manipulating large data sets or you need to perform intensive computations, you are likely to get a very poor performance. If you have Matlab applications that could benefit from a higher performance or parallel execution, please contact us and let us know about it - there are now some third party applications that might make Matlab much more efficient on COSMOS, but we would need to evaluate them with real applications in order to get a realistic idea of its usefulness.

Contents