To activate env:
source activate pyTFenv
To deactivate:
source datctivate
Jun 27, 2019
Jun 18, 2019
Create python virutal environment for ML
unloaded existing python module - as I had already python 3.7 by now)
look for all python modules
load python/3.7.0-anaconda3-5.3.0
Update conda
Then activate the environment:
source activate MLenv
Now we have our python 3.7 environment
We can install all the frameworks we need here.
To deactivate the environment,
source deactivate
.
.
.
.
.
.
.
.
I figured out that TensorFlow 1.12.0 only works with Python version 3.5.2. I had Python 3.7 but that didn't work. So, I had to downgrade Python and then I could install TensorFlow to make it work.
Environment location:
/ccs/home/uma/.conda/envs/pyTFptEnv
look for all python modules
load python/3.7.0-anaconda3-5.3.0
$ conda -VUpdate conda
conda update condaconda create -n MLenv python=3.7 anacondaThen activate the environment:
source activate MLenv
Now we have our python 3.7 environment
We can install all the frameworks we need here.
To deactivate the environment,
source deactivate
.
.
.
.
.
.
.
.
6. Delete a no longer needed virtual environment
- To delete a conda environment, enter the following, where yourenvname is the name of the environment you wish to delete.
conda remove -n yourenvname -all
I figured out that TensorFlow 1.12.0 only works with Python version 3.5.2. I had Python 3.7 but that didn't work. So, I had to downgrade Python and then I could install TensorFlow to make it work.
To downgrade your python version from 3.7 to 3.6
conda install python=3.6.8Environment location:
/ccs/home/uma/.conda/envs/pyTFptEnv
Mar 1, 2019
Reviving Aria 10 work on Oswald03
xxk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul/bin$ env CL_CONTEXT_E
MULATOR_DEVICE_INTELFPGA=1 ./matmul_ACC
CPU Elapsed time = 0.392515 sec
Accelerator Elapsed time = 22.417378 sec
Verification Successful err = 0.000000e+00
On Oswald >
matmul example
change the makefile
16:52:09 udk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul/bin$ ls
__all_sources.cl matmul_ACC openarc_kernel_EmulatorDeviceEmulatedDevice.aocx
binBuilder_opencl openarc_kernel.cl Timer
16:53:03 udk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul/bin$ cd ..
16:53:06 udk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul$ cd cetus_output/
16:53:31 udk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul/cetus_output$ ls
matmul.cpp openarc_kernel openarc_kernel.aoco openarc_kernel.aocx openarc_kernel.cl
16:53:32 udk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul/cetus_output$ aoc -v openarc_kernel.cl -o ../bin/openarc_kernel_p510t_sch_ax115PCIe510n.aocx --board p510t_sch_ax115
Warning: Please use -board= instead of --board
aoc: Environment checks are completed successfully.
aoc: If necessary for the compile, your BAK files will be cached here: /var/tmp/aocl/
You are now compiling the full flow!!
aoc: Selected target board p510t_sch_ax115
aoc: Running OpenCL parser....
aoc: OpenCL parser completed successfully.
aoc: Optimizing and doing static analysis of code...
aoc: Linking with IP library ...
Checking if memory usage is larger than 100%
aoc: First stage compilation completed successfully.
Compiling for FPGA. This process may take a long time, please be patient.
MULATOR_DEVICE_INTELFPGA=1 ./matmul_ACC
CPU Elapsed time = 0.392515 sec
Accelerator Elapsed time = 22.417378 sec
Verification Successful err = 0.000000e+00
On Oswald >
matmul example
change the makefile
16:52:09 udk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul/bin$ ls
__all_sources.cl matmul_ACC openarc_kernel_EmulatorDeviceEmulatedDevice.aocx
binBuilder_opencl openarc_kernel.cl Timer
16:53:03 udk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul/bin$ cd ..
16:53:06 udk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul$ cd cetus_output/
16:53:31 udk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul/cetus_output$ ls
matmul.cpp openarc_kernel openarc_kernel.aoco openarc_kernel.aocx openarc_kernel.cl
16:53:32 udk@oswald03:~/gamesskernel/openarcOS/test/examples/openarc/matmul/cetus_output$ aoc -v openarc_kernel.cl -o ../bin/openarc_kernel_p510t_sch_ax115PCIe510n.aocx --board p510t_sch_ax115
Warning: Please use -board=
aoc: Environment checks are completed successfully.
aoc: If necessary for the compile, your BAK files will be cached here: /var/tmp/aocl/
You are now compiling the full flow!!
aoc: Selected target board p510t_sch_ax115
aoc: Running OpenCL parser....
aoc: OpenCL parser completed successfully.
aoc: Optimizing and doing static analysis of code...
aoc: Linking with IP library ...
Checking if memory usage is larger than 100%
aoc: First stage compilation completed successfully.
Compiling for FPGA. This process may take a long time, please be patient.
Feb 5, 2019
Jan 17, 2019
Creating OpenACC C++ libraries
pgi/15.10.0 supports pgCC for c++
Newer pgi/18 doesn't recognize pgCC on Titan
A good read
http://www.drdobbs.com/parallel/creating-and-using-libraries-with-openac/240012502?pgno=2
https://kristerw.blogspot.com/2017/04/building-gcc-with-support-for-nvidia.html
acc benchmark code
https://github.com/EPCCed/epcc-openacc-benchmarks/blob/master/
OpenACC code targeted at an Nvidia GPU must be compiled with the PGI compiler using at least the following options:
pgcc source_code.c -ta=nvidia -acc
scc1% pgcc -o mycode -acc -Minfo mycode.c
In the above example,
-acc turns on the OpenACC feature while -Minfo returns additional information on the compilation.pgi/15.10.0 supports pgCC for c++
Newer pgi/18 doesn't recognize pgCC on Titan
A good read
http://www.drdobbs.com/parallel/creating-and-using-libraries-with-openac/240012502?pgno=2
https://kristerw.blogspot.com/2017/04/building-gcc-with-support-for-nvidia.html
acc benchmark code
https://github.com/EPCCed/epcc-openacc-benchmarks/blob/master/
Dec 8, 2018
Get rid of ^M from Visual Studio to Vim
:e ++ff=dos
The :e ++ff=dos command tells Vim to read the file again, forcing dos file format. Vim will remove CRLF and LF-only line endings, leaving only the text of each line in the buffer.
then
:set ff=unix
and finally
:wq Oct 22, 2018
Proxy Setup
HF and DFT calculations background theory:
http://www.turbomole-gmbh.com/manuals/version_6_4/Documentation_html/node130.html
In simint ostei_config.h
has Maxam and maxder configuration set to 1 and 0 by me: THis might not make sense. Verify!
1- build simint
2-build oed
3-build simgms
4-create main object file in Debug
5-build with main and create main executable
Newark modules from earlier
module load gcc/5.3.0
module load make
module load python/2.7.11
module load openmpi/gnu/1.10.4-gnu
module load hdf5/1.8.10-intel
module load boost/1.60.0-gnu5
module load mkl/2013.5.192
module load mpich/3.1.2-gnu
module load cuda/8.0
module load cmake
copied simgms to titan:
scp main.cpp uma@titan.ccs.ornl.gov:/ccs/home/uma/fall18/gamess/
copied two versions of simgms from the old directory to /home/udk/proxy
changed the modules to match old modules: also use the same games_env environment.
download simint from http://www.bennyp.org/research/simint/
Simint is a vectorized implementation of Obara saika method for calculating electron repulsion integrals.
Speedup is gotten from vectorizing primitive loop of the OS algorithm.
Additional vectorizations are left to the compiler.
rename the exatracted file to simint
udk@oswald02:~/proxy/simgms$ cd simint
udk@oswald02:~/proxy/simgms/simint$ ls
CHANGELOG cmake CMakeLists.txt examples LICENSE README simint test
/home/udk/proxy/simgms/simint>> CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=scalar -DCMAKE_PREFIX_PATH=/home/udk/proxy/simint -DCMAKE_INSTALL_LIBDIR=/home/udk/proxy/simgms/simint/lib -DCMAKE_INSTALL_INCLUDEDIR=/home/udk/proxy/simgms/simint/include -DCMAKE_INSTALL_PREFIX=/home/udk/proxy/simgms/simint
" " >> make -j8
>> makde all
"" >> make install DESTDIR=.
This installs everything.
************************************************
udk@oswald02:~/proxy/simgms/simint$ make -j8
[100%] Linking CXX executable test_eri_deriv1
[100%] Linking CXX executable screentest
[100%] Linking CXX executable test_eri
[100%] Built target benchmark_eri
[100%] Built target test_eri_deriv1
[100%] Built target screentest
[100%] Built target test_eri
udk@oswald02:~/proxy/simgms/simint$ make all
[ 99%] Built target simint
[ 99%] Built target test_common
[ 99%] Built target test_eri
[100%] Built target test_eri_deriv1
[100%] Built target benchmark_eri
[100%] Built target screentest
[100%] Built target example_packed
[100%] Built target example1
************************************************
on TITAN
Follow the same instructions.
We downgraded to gcc 5.3 and upgraded to cmake3
Also loaded module intel.
CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=scalar -DCMAKE_PREFIX_PATH=/ccs/home/uma/fall18/simgms/simgms/simint -DCMAKE_INSTALL_LIBDIR=/ccs/home/uma/fall18/simgms/simgms/simint/lib -DCMAKE_INSTALL_INCLUDEDIR=/ccs/home/uma/fall18/simgms/simgms/simint/include -DCMAKE_INSTALL_PREFIX=/ccs/home/uma/fall18/simgms/simgms/simint
************************************************************************
University of Florida oed library (fortran)
https://github.com/hpcgarage/OptErd/tree/master/oed
map the correct gcc in exl.
Compiling simgms
Had to change paths
in simint_eri.h =>
#ifdef __cplusplus
#include "cpp_restrict.hpp"
THere are 2 files that are .in which has to converted to .h ,
in vectorization/vector_config.h we need to add the vector type as below
/*#define SIMINT_AVX*/
#define SIMINT_SCALAR
In ostei/ostei_config.h
It seems to read duplicate simint files; one from the include and one from the other; Maybe we need to get rid of include dir.
or delete the other directory;
one in the include directory had Maxam and maxder set to 7 and 0 while the other path /simint/simint/ostei looks like just the source. Why is this included then?
INCLUDED HEADERS OF SIMINT ARE:
/proxy/simgms/simint/simint/
We need to include simint/simint as an include directory for building simgms.
This has to be not hardcoded; but set as variables for simint in the app.
defined SIMINT_OSTEI_MAX_WORKMEM as 1024*1024 in simgms_tei.cpp because it SIMINT_OSTEI_MAX_WORKMEM is not defined in line 66 .
They might have set this in a makefile. But does this make sense?
copied the main.cpp file to proxy/simgms location from oldhome's simgmsCoAC
removed define c version from file.
/proxy/simgms/Debug$ g++ -c ../main.cpp
This creates the object file in Debug folder.
Copied simint from /oldhome/umunipala/gamess/gamess-kris-simgms/simgms
''''
Removed the include siminit from the subdir.mk in Debug path. -I/home/udk/proxy/simgms/simint/include -
Create main executable.
Debug>>
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/udk/proxy/simgms/Debug -lsimgms -L/home/udk/proxy/simgms/simint/lib -lsimint -L/home/udk/proxy/simgms/oed -loed -loedlib -L/home/udk/intel/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/udk/proxy/simgms/simint/simint/include -I/home/udk/proxy/simgms/include -I/home/udk/intel/mkl/include -Wl,--end-group -ldl -lpthread -lm
----------------22 oct_________________________
udk@oswald02:~/proxy/simgms/Debug$ ./main
./main: error while loading shared libraries: libmkl_mc.so: cannot open shared object file: No such file or directory
it is in mkl libraries
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/udk/intel/mkl/lib/intel64
executable runs now.
Look at the code :: algorithm failed to compute eigenvalues.
Basis sets:
http://www.turbomole-gmbh.com/manuals/version_6_4/Documentation_html/node130.html
In simint ostei_config.h
has Maxam and maxder configuration set to 1 and 0 by me: THis might not make sense. Verify!
1- build simint
2-build oed
3-build simgms
4-create main object file in Debug
5-build with main and create main executable
Newark modules from earlier
module load gcc/5.3.0
module load make
module load python/2.7.11
module load openmpi/gnu/1.10.4-gnu
module load hdf5/1.8.10-intel
module load boost/1.60.0-gnu5
module load mkl/2013.5.192
module load mpich/3.1.2-gnu
module load cuda/8.0
module load cmake
copied simgms to titan:
scp main.cpp uma@titan.ccs.ornl.gov:/ccs/home/uma/fall18/gamess/
copied two versions of simgms from the old directory to /home/udk/proxy
changed the modules to match old modules: also use the same games_env environment.
download simint from http://www.bennyp.org/research/simint/
Simint is a vectorized implementation of Obara saika method for calculating electron repulsion integrals.
Speedup is gotten from vectorizing primitive loop of the OS algorithm.
Additional vectorizations are left to the compiler.
rename the exatracted file to simint
udk@oswald02:~/proxy/simgms$ cd simint
udk@oswald02:~/proxy/simgms/simint$ ls
CHANGELOG cmake CMakeLists.txt examples LICENSE README simint test
/home/udk/proxy/simgms/simint>> CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=scalar -DCMAKE_PREFIX_PATH=/home/udk/proxy/simint -DCMAKE_INSTALL_LIBDIR=/home/udk/proxy/simgms/simint/lib -DCMAKE_INSTALL_INCLUDEDIR=/home/udk/proxy/simgms/simint/include -DCMAKE_INSTALL_PREFIX=/home/udk/proxy/simgms/simint
" " >> make -j8
>> makde all
"" >> make install DESTDIR=.
This installs everything.
************************************************
udk@oswald02:~/proxy/simgms/simint$ make -j8
[100%] Linking CXX executable test_eri_deriv1
[100%] Linking CXX executable screentest
[100%] Linking CXX executable test_eri
[100%] Built target benchmark_eri
[100%] Built target test_eri_deriv1
[100%] Built target screentest
[100%] Built target test_eri
udk@oswald02:~/proxy/simgms/simint$ make all
[ 99%] Built target simint
[ 99%] Built target test_common
[ 99%] Built target test_eri
[100%] Built target test_eri_deriv1
[100%] Built target benchmark_eri
[100%] Built target screentest
[100%] Built target example_packed
[100%] Built target example1
************************************************
on TITAN
Follow the same instructions.
We downgraded to gcc 5.3 and upgraded to cmake3
Also loaded module intel.
CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=scalar -DCMAKE_PREFIX_PATH=/ccs/home/uma/fall18/simgms/simgms/simint -DCMAKE_INSTALL_LIBDIR=/ccs/home/uma/fall18/simgms/simgms/simint/lib -DCMAKE_INSTALL_INCLUDEDIR=/ccs/home/uma/fall18/simgms/simgms/simint/include -DCMAKE_INSTALL_PREFIX=/ccs/home/uma/fall18/simgms/simgms/simint
************************************************************************
University of Florida oed library (fortran)
https://github.com/hpcgarage/OptErd/tree/master/oed
map the correct gcc in exl.
Compiling simgms
Had to change paths
in simint_eri.h =>
#ifdef __cplusplus
#include "cpp_restrict.hpp"
THere are 2 files that are .in which has to converted to .h ,
in vectorization/vector_config.h we need to add the vector type as below
/*#define SIMINT_AVX*/
#define SIMINT_SCALAR
In ostei/ostei_config.h
It seems to read duplicate simint files; one from the include and one from the other; Maybe we need to get rid of include dir.
or delete the other directory;
one in the include directory had Maxam and maxder set to 7 and 0 while the other path /simint/simint/ostei looks like just the source. Why is this included then?
INCLUDED HEADERS OF SIMINT ARE:
/proxy/simgms/simint/simint/
We need to include simint/simint as an include directory for building simgms.
This has to be not hardcoded; but set as variables for simint in the app.
defined SIMINT_OSTEI_MAX_WORKMEM as 1024*1024 in simgms_tei.cpp because it SIMINT_OSTEI_MAX_WORKMEM is not defined in line 66 .
They might have set this in a makefile. But does this make sense?
copied the main.cpp file to proxy/simgms location from oldhome's simgmsCoAC
removed define c version from file.
/proxy/simgms/Debug$ g++ -c ../main.cpp
This creates the object file in Debug folder.
Copied simint from /oldhome/umunipala/gamess/gamess-kris-simgms/simgms
''''
Removed the include siminit from the subdir.mk in Debug path. -I/home/udk/proxy/simgms/simint/include -
Create main executable.
Debug>>
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/udk/proxy/simgms/Debug -lsimgms -L/home/udk/proxy/simgms/simint/lib -lsimint -L/home/udk/proxy/simgms/oed -loed -loedlib -L/home/udk/intel/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/udk/proxy/simgms/simint/simint/include -I/home/udk/proxy/simgms/include -I/home/udk/intel/mkl/include -Wl,--end-group -ldl -lpthread -lm
----------------22 oct_________________________
udk@oswald02:~/proxy/simgms/Debug$ ./main
./main: error while loading shared libraries: libmkl_mc.so: cannot open shared object file: No such file or directory
it is in mkl libraries
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/udk/intel/mkl/lib/intel64
executable runs now.
Look at the code :: algorithm failed to compute eigenvalues.
Basis sets:
https://bse.pnl.gov/bse/portal
Feb 14, 2018
c and others - deep learning benchmarks
yusugomori (in C)
https://github.com/yusugomori/DeepLearning/blob/master/c/DBN.c
baidu research
https://github.com/baidu-research/DeepBench
libdeep c/c++ deep learning library
https://github.com/bashrc/libdeep
caffe
http://caffe.berkeleyvision.org/
convnet-benchmarks
https://github.com/soumith/convnet-benchmarks
Dec 5, 2017
SIMGMS on GPU
Currently Loaded Modulefiles:
1) modules 7) gcc/5.3.0 13) m4/1.4.15
2) git/2.10.1 8) cuda/8.0 14) boost/1.60.0-gnu5
3) gmp/5.0.5 9) make/4.1 15) mkl/2013.5.192
4) mpfr/3.1.0 10) openssl/1.0.2c 16) mpich/3.1.2-gnu
5) mpc/0.9 11) python/2.7.11 17) cmake/3.3.2
6) isl/0.14 12) hdf5/1.8.10-intel 18) openmpi/1.10.0-gnu
change main and
simgmsCoAC/Debug>> g++ -c ../main.cpp
create the main.o
To run the code in a single GPU
CUDA_VISIBLE_DEVICES=1
1) modules 7) gcc/5.3.0 13) m4/1.4.15
2) git/2.10.1 8) cuda/8.0 14) boost/1.60.0-gnu5
3) gmp/5.0.5 9) make/4.1 15) mkl/2013.5.192
4) mpfr/3.1.0 10) openssl/1.0.2c 16) mpich/3.1.2-gnu
5) mpc/0.9 11) python/2.7.11 17) cmake/3.3.2
6) isl/0.14 12) hdf5/1.8.10-intel 18) openmpi/1.10.0-gnu
change main and
simgmsCoAC/Debug>> g++ -c ../main.cpp
create the main.o
To run the code in a single GPU
CUDA_VISIBLE_DEVICES=1
Enable double precision
OpenARC will add the following directive at the beginning of the generated OpenCL kernel file.
OpenARC will add the following directive at the beginning of the generated OpenCL kernel file.
#pragma OPENCL EXTENSION cl_khr_fp64 : enable
Nov 25, 2017
Using openARC for CUDA GPUs
matmul OpenACC-CUDA example:
export OMP_NUM_THREADS=2
export OMP_NUM_THREADS=2
./matmul_CUDA
Number of OpenMP threads 2
CPU Elapsed time = 0.181340 sec
Accelerator Elapsed time (CUDA) = 0.006136 sec
Accelerator Elapsed time (OpenACC) = 0.006496 sec
Verification Successful err = 0.000000e+00
Nov 6, 2017
FPGA power measurements
Compile on Oswald00
./O2script
make
Then generate hardware on Oswald03
If you goto
/opt/altera/17.0/quartus/bin run the script quartus_pow from there on top.qpf
-------------------------------------------------------------------------------------------------------------------
./quartus_pow /home/umunipala/openarc1/test/examples/openarc/matmul/cetus_output/openarc_kernel/top.qpf
./O2script
make
Then generate hardware on Oswald03
If you goto
/opt/altera/17.0/quartus/bin run the script quartus_pow from there on top.qpf
-------------------------------------------------------------------------------------------------------------------
./quartus_pow /home/umunipala/openarc1/test/examples/openarc/matmul/cetus_output/openarc_kernel/top.qpf
Aug 25, 2017
Run example matmul on FPGA
Don't change the makefile. Default is on device
1. Run script O2build.script
This creates the host code and the kernel
To run on device:
Makefile flags
OMP ?= 0
MODE ?= normal
AOCL_BOARD ?= p385_hpc_d5
#AOCL_BOARD ?= p385_hpc_a7
#AOCL_FLAGS ?= -march=emulator
#AOCL_FLAGS ?= -march=emulator -c
AOCL_FLAGS ?= -v -c --report
#AOCL_FLAGS ?= --report
2. Run make
This creates the aoc file
After this bin and cetus look like this
[umunipala@newark cetus_output]$ ls
matmul.cpp openarc_kernel openarc_kernel.aoco openarc_kernel.cl
[umunipala@newark cetus_output]$ cd ../bin
[umunipala@newark bin]$ ls
Timer binBuilder_opencl matmul_ACC openarc_kernel.cl
now goto cetus output > (where aoc file is)
aoc openarc_kernel.cl -o openarc_kernel_p385_hpc_d5PCIe385n.aocx --board p385_hpc_a7 - here a7 is the second board; need to correct to a5
(on oswald
aoc -v openarc_kernel.cl -o ../bin/openarc_kernel_p510t_sch_ax115PCIe510n.aocx --board p510t_sch_ax115
)
about 2+ hours later...
goto cetus_output>>
Then run ./SIMGMS_ACC this takes device in default.
#################################################################################Running on FPGA device on oswald??
1. create the aocx from aoco first
aoc -v openarc_kernel.cl -o ../bin/openarc_kernel_p510t_sch_ax115PCIe510n.aocx --board p510t_sch_ax115
Then run ./SIMGMS_ACC this takes device in default.
Don't change the makefile. Default is on device
1. Run script O2build.script
This creates the host code and the kernel
To run on device:
Makefile flags
OMP ?= 0
MODE ?= normal
AOCL_BOARD ?= p385_hpc_d5
#AOCL_BOARD ?= p385_hpc_a7
#AOCL_FLAGS ?= -march=emulator
#AOCL_FLAGS ?= -march=emulator -c
AOCL_FLAGS ?= -v -c --report
#AOCL_FLAGS ?= --report
This creates the aoc file
After this bin and cetus look like this
[umunipala@newark cetus_output]$ ls
matmul.cpp openarc_kernel openarc_kernel.aoco openarc_kernel.cl
[umunipala@newark cetus_output]$ cd ../bin
[umunipala@newark bin]$ ls
Timer binBuilder_opencl matmul_ACC openarc_kernel.cl
now goto cetus output > (where aoc file is)
aoc openarc_kernel.cl -o openarc_kernel_p385_hpc_d5PCIe385n.aocx --board p385_hpc_a7 - here a7 is the second board; need to correct to a5
This is similar to
(Compile kernel for emulation)
aoc -v --board device/hello_world.cl -o bin/hello_world.aocx ( )
aoc -march=emulator -v --board
device/hello_world.cl -o bin/hello_world_emulation.aocx
aoc -v openarc_kernel.cl -o ../bin/openarc_kernel_p385_hpc_d5PCIe385n.aocx --board p385_hpc_d5
(on oswald
aoc -v openarc_kernel.cl -o ../bin/openarc_kernel_p510t_sch_ax115PCIe510n.aocx --board p510t_sch_ax115
)
goto cetus_output>>
Then run ./SIMGMS_ACC this takes device in default.
#################################################################################Running on FPGA device on oswald??
1. create the aocx from aoco first
aoc -v openarc_kernel.cl -o ../bin/openarc_kernel_p510t_sch_ax115PCIe510n.aocx --board p510t_sch_ax115
Then run ./SIMGMS_ACC this takes device in default.
Aug 24, 2017
SIMGMS on FPGA
---------------------7/27 running example of openarc
export openarc=/home/umunipala/openARC
export OPENARC_ARCH=3
/home/umunipala/openARC/bin/openarc matmul.c
2days july last week for converting to openacc and learning about openarc
8-07-2017---------------------------------------
reading about openacc : still translating to openACC
run gprof: not necessory
----8-9-2017 still trying to figure out how to correctly annotate openarc and give options correct.
/home/umunipala/openARC/bin/openarc -AccPrivatization=1 AccReduction=1 macro=_N_=__inputSize__,VERIFICATION=1 addIncludePath=__openarcrt__ assumeNoAliasingAmongKernelArgs /home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simgms_scf__c.c >& openARCout
--------------------------------8-14-2017
We are not touching simgms directory.
All changes with openacc and openarc are in this directory below.
/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC>>
changed paths in subdir.mk and changed for main.c
changed main.cpp into main.c
complains about extern: so commented it
generated libsimgms
copy lulesh makefile and build scritpt into simgmsCoAC and change paths to make.template and make.header
---------------------------------------------------------------------
now compile c programs and get main executablell
gcc -Wl,--start-group -Wall -g -O -O1 -o main -lgfortran -fopenmp -lgomp ../main.c -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
got errros::
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.c -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.c -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
-------------------------------------------------------------------
error compiling.
[umunipala@newark Debug]$ g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.c -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
/tmp/ccYSzEib.o: In function `main':
/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/Debug/../main.c:35: undefined reference to `simgms_scf_c(double const*, double const*, double const*, double const*, double const*, double const*, double const*, double const*, long const*, long const*, long const*, long const*, long const*, long, long, long, long, long, double const*, double const*, double)'
For openarc we need to add the correct files and modify makefiles.
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.o -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
------------------------8-15-2017
openarc supports ansi c older than 99
need to remove // comments 11 to 12.21
------------08-23 Try to recreate fpga file structure in simgms dir
/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgmsCoAC/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
-------------Aug 24----------------------
rebuilt simint and simgms
can not generate make executable simgms_c throws an error.
reason: we changed simgms_scf for openacc. when openacc converted the simgms c file to cpp file it needed different path + to define extern c in a few files.
1. simgms_scf.h
2.simgms_init.h and some more (around 2 3 more - Forgot! Work from saturday, Aug 22)
main.o has to be in Debug directory.
Realized that we do not need the main executable anymore. If we need that it is already there in simgms path.
Undoing and trying to run the exe with openarc as before.
For some reason I again got the illegal instruction.
When I looked at simint library it was an old one. So recompiled simint.
Then copy the library to simint/simint/lib path : this is what is in OpenARC makefile settings for Simgms project in simgmsCoAC directory.
This should fix the error.
Then proceed to adding more openACC
then add timers
Aug 11, 2017
Executing FPGA host code
-------------------------------------- 08-11-17
1161 aoc --list-boards
1172 aoc openarc_kernel.cl -o openarc_kernel_p385_hpc_d5PCIe385n.aocx --board p385_hpc_a7
Running aocx FPGA executable
Need to know the linking command
[umunipala@newark matmul]$ aocl linkflags
-L/opt/altera/15.0/hld/board/nalla_pcie/linux64/lib -L/opt/altera/16.1/standard/hld/host/linux64/lib -Wl,--no-as-needed -lalteracl -lalterahalmmd -lnalla_pcie_mmd -lelf
Those are needed only to generate the executable.
Running:
env CL_CONTEXT_EMULATOR_DEVICE_ALTERA=1 ./matmul_ACC
This uses the fpga executable (aocx file) in the current library.
HELP
https://www.altera.com/documentation/mwh1391807309901.html#ewa1435154782031
1161 aoc --list-boards
1172 aoc openarc_kernel.cl -o openarc_kernel_p385_hpc_d5PCIe385n.aocx --board p385_hpc_a7
Running aocx FPGA executable
Need to know the linking command
[umunipala@newark matmul]$ aocl linkflags
-L/opt/altera/15.0/hld/board/nalla_pcie/linux64/lib -L/opt/altera/16.1/standard/hld/host/linux64/lib -Wl,--no-as-needed -lalteracl -lalterahalmmd -lnalla_pcie_mmd -lelf
Those are needed only to generate the executable.
Running:
env CL_CONTEXT_EMULATOR_DEVICE_ALTERA=1 ./matmul_ACC
This uses the fpga executable (aocx file) in the current library.
HELP
https://www.altera.com/documentation/mwh1391807309901.html#ewa1435154782031
Jul 21, 2017
Upto compiling and linking main executable of sim
and add the lines below (logged into excl not newark yet.
. /opt/shared/sw/x86_64-unknown-linux-gnu/modules-tcl85/default/init/bash
module load modules
module use /opt/shared/sw/x86_64-unknown-linux-gnu/modulefiles/rhel6
#module use /opt/shared/sw/x86_64-unknown-linux-gnu/modulefiles/rhel7
then source or login again
. ~/.bashrc - sourcing bashrc: need to be in bash shell for this command
doesn't work for tcsh. no modules available in tcsh? why?
****If you are out of newark and load modules when u are in newark the paths are still there.Run config file in tcsh this way ; else it can't find modules
modules loaded in newark
d Modulefiles:
1) modules 3) boost/1.54.0-intel 5) python/2.7.11 7) hdf5/1.8.10-intel
2) m4/1.4.15 4) openssl/1.0.2c 6) openmpi/1.10.1-gnu
set env variables for games
export GMS_BUILD_DIR='/home/umunipala/gamess/gamess-kris-simgms
'
export GMS_TARGET=''
export GMS_PATH='/home/umunipala/gamess/gamess-kris-simgms
'
export GMS_FORTRAN='ifort'
export GMS_MATHLIB=''
set GMS_DDI_COMM=''
export GMS_TARGET=linux64
export GMS_PATH=/home/umunipala/gamess/gamess-kris-simgms
check if shell is interactive
[[ $- == *i* ]] && echo 'Interactive' || echo 'Not interactive'
----------------------------------------------4/13
Currently Loaded Modulefiles:
Currently Loaded Modulefiles: openssl is loaded by openmpi
1) modules 4) hdf5/1.8.10-intel 7) boost/1.60.0-gnu5
2) openssl/1.0.2c 5) openmpi/1.10.1-gnu 8) mkl/2013.5.192
3) python/2.7.11 6) m4/1.4.15 9) intel/2013.5.192
RUNNING CONFIG
>tcsh
>source ./config
Errors: gcc gfortran didn't show up . Logged in and out and its shown in
[umunipala@newark ~]$ which gcc
/usr/lib64/ccache/gcc
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)
Alas, your version of gfortran does not support REAL*16,
so relativistic integrals cannot use quadruple precision.
Other than this, everything will work properly.
MKL configuration
MKL version (or 'proceed')? proceed
Math library 'mkl' will be taken from /opt/intel/mkl/lib/intel64
---------------------------
Currently Loaded Modulefiles:
1) modules 6) hdf5/1.8.10-intel
2) mkl/2013.5.192 7) m4/1.4.15
3) intel/2013.5.192 8) boost/1.54.0-intel
4) openssl/1.0.2c 9) openmpi/1.10.1-gnu
5) python/2.7.11 10) mpich/3.1.2-gnu
mpich path::
set to::
/opt/shared/sw/x86_64-unknown-linux-gnu/mpich/3.1.2-gnu-rhel6
bin is here.
5) The subdirectory gamess/ddi contains a script 'compddi' to compile
the computer science support for parallel GAMESS, known as the
Distributed Data Interface (DDI).
There is one possible customization of 'compddi' that must be done
by hand-editing 'compddi', prior to running it. If your system's
number of cores per node or total number of nodes is large, reset
the defaults of MAXCPUS=8 and MAXNODES=128.
>did not change coz its only 8
>compddi : got error due to mpi
/opt/shared/sw/x86_64-unknown-linux-gnu/mpich/include/mpi.h
fixed it by changing mpi path to this(above)
compiled::
games>./compall
Thu Apr 13 15:17:10 EDT 2017
0.063u 0.018s 0:00.10 70.0% 0+0k 16+24io 0pf+0w
------------------- done with all compilations --------------------
linking failure : missing simgms.o
------------------4-17
**./comp simgms add line to compall
>./compal : no errors.
goto games/machines>
look in readme.unix step 7
goto games> run lked script to link libraries
./lked gamess 01 >& lked.log &
** change set SIMGMSLIBS line in lked script (this is not set for current user;hardcoded.)
set SIMGMSLIBS = "-L/home/kris/gamess/simgms/Debug -lsimgms -lstdc++ -L/home/kris/gamess/simgms/simint/lib -lsimint"
change the above to:
set SIMGMSLIBS = "-L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -lstdc++ -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/lib -lsimint"
/home/umunipala/gamess/gamess-kris-simgms/simgms>>>
Readme says,
Add simgms.src to gamess source directory and use modified lked to link with gamess binary
/home/umunipala/gamess/gamess-kris-simgms/simgms/gamess_files>>
there is a lked and simgms.src
lked looks like the original config script
*****************************************Above might not be necessory
move simgms to source directory.
goto:
/home/umunipala/gamess/gamess-kris-simgms/simgms/gamess_files>>
cp simgms.src ../../source/simgms.src
ran the lked in this directory: asks to run config.
replaced original lked in games>> same error
./lked gamess 01 >& lked.log &
run /.compal
_____________________________________________________4/18 changed mkl path in config
lked errors, makefile in simgms/debug had licencing errors with intel
>Going back to configuring with gcc because the makefile is autogenerated.
/opt/intel/composer_xe_2013.5.192/mkl
changed modules
in readme.unix that is in machines folder says
If you are using the Ubuntu version of Linux, you will get better results by changing the top line of each script to read
#!/bin/tcsh in all the scripts below:
config compall comp lked rungms runall
run the compddi in gamess/ddi
run compall after going back to bash and loading modules.
reverted from tcsh to bash and changed to #!/bin/csh in compall
goto games/simgms/Debug> make -icpc=cc
ran ./lked gamess 01 >& lked.log & and still there is no libsimgms.a or .so
checked with find . -name libsimgms*
not available.
module load gcc
module load make
module load python/2.7.11
module load openmpi/1.10.1-gnu
module load hdf5/1.8.10-intel
module load boost/1.60.0-gnu5
module load mkl/2013.5.192
module load mpich/3.1.2-gnu
____________________4/24/17
changing makefiles in Debug folder
subdir.mk : contains wrong path
/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 is mkl lib and they are shared/dynamic
-std=gnu++11 -lmkl_intel_ilp64 -lmkl_gnu_thread
/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
setenv LD_LIBRARY_PATH /opt/intel/composer_xe_2013.5.192/mkl/lib/intel64:$LD_LIBRARY_PATH
-----------------------------4/26---------
fix intrinsics_avx.h error
________________________-approach 1--didn't work
goto /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/vectorization
>>
current files here are:
intrinsics_avx512.h intrinsics_scalar.h vectorization.h
intrinsics_avx.h intrinsics_sse.h
>> Now adding headers..
change the header from <> to "immintrin.h" in intrinsics_avx.h
>>gcc libs
/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib/gcc/x86_64-unknown-linux-gnu/5.3.0/include
-------------------------------approach 2-----------------
define your own macro like this in intrinsics_avx.h
#define _mm256_set_m128d(vh, vl) \
_mm256_insertf128_pd(_mm256_castpd128_pd256(vl), (vh), 1)
Instead #define SIMINT_GCC in intrinsics_avx.h
------------------------------------------------------------make o/p
[umunipala@newark Debug]$ make
Building file: ../simgms_init.cpp
Invoking: G++ Compiler
g++ -g -O -O1 -std=c++11 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -libmkl_lapack95_lp64 -libmkl_intel_lp64 -libmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -DDEBUG -DMKL_ILP64 -MMD -MP -MF"simgms_init.d" -MT"simgms_init.d" -c -o "simgms_init.o" "../simgms_init.cpp"
In file included from /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/incl
ude/simint/vectorization/vectorization.h:11:0,
from /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/incl
ude/simint/ostei/ostei_config.h:3,
from /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/incl
ude/simint/simint.h:5,
from ../simgms_init.h:3,
from ../simgms_init.cpp:3:
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/vectorizati
on/intrinsics_avx.h: In function '__m256d simint_exp_vec4(__m256d)':
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/vectorizati
on/intrinsics_avx.h:25:48: warning: AVX vector return without AVX enabled changes the ABI [-Wpsabi]
static inline __m256d simint_exp_vec4(__m256d x)
^
Finished building: ../simgms_init.cpp
Building file: ../simgms_math.cpp
Invoking: G++ Compiler
g++ -g -O -O1 -std=c++11 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -libmkl_lapack95_lp64 -libmkl_intel_lp64 -libmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -DDEBUG -DMKL_ILP64 -MMD -MP -MF"simgms_math.d" -MT"simgms_math.d" -c -o "simgms_math.o" "../simgms_math.cpp"
../simgms_math.cpp:2:25: fatal error: mkl_lapacke.h: No such file or directory
compilation terminated.
---------------------------------
/opt/intel/composer_xe_2013.5.192/mkl????????????????? sometimes this path shows as available and sometimes not? why?
/opt/intel/composer_xe_2013.5.192/mkl/include
has mkl_lapack.h file
-----------------------------------------------------------
generated libsimint.a in libs and there is two more files simgms_init.d and simgms_init.o generated.
compiling lked script in main directory
./lked gamess 01 >& lked.log &
-------------------------------------------------current build error
gcc -mavx -g -O -O1 -std=c++11 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -libmkl_mc -libmkl_sequential -libmkl_gf_ilp64 -libmkl_avx -libmkl_lapack95_lp64 -libmkl_intel_lp64 -libmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -DDEBUG -DMKL_ILP64 -MMD -MP -MF"simgms_math.d" -MT"simgms_math.d" -c -o "simgms_math.o" "../simgms_math.cpp"
In file included from ../simgms_math.cpp:2:0:
../mkl_lapacke.h:32:23: fatal error: mkl_types.h: No such file or directory
compilation terminated.
subdir.mk:31: recipe for target 'simgms_math.o' failed
make: *** [simgms_math.o] Error 1
-------------------------------------
simgms/Debug>
got an error 'CBLAS_LAYOUT undeclared
fixed it with adding line typedef CBLAS_ORDER CBLAS_LAYOUT; /* this for backward compatibility with CBLAS_ORDER */
to simgms_math.cpp
Ran make in Debug folder.
This generated libsimgs.a
go up to main lked script
________________________________modified headers in simgms_math.cpp
added include files.
/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64
./lked gamess 01 >& lked_may_1.log &
libs gcc
/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib64
added this line to simint.
define
look for files in /opt/intel/composer_xe_2013.5.192/compiler/lib/intel64
gcc -I/opt/intel/composer_xe_2013.5.192/compiler/include-L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 __intel_ssse3_rep_memcpy
-----
added macro #define SIMINT_GC in simgms_init.cpp and simgms_oei.cpp --- not sure: bcoz it was mentioned in vectorization.h
---------------------may 1
copied all the header files from the Debug directory generated from the files like simgms_math, simgms_oie etc (which we in Debug )into the folder below.
Then added the bath in subdir.mk
-I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/debug
./lked gamess 01 >& lked_may_1.log &
*** Run makefile from following directory:
/home/umunipala/gamess/gamess-kris-simgms/simgms/oed
changed ifort to gfortran in Makefile
FC=gfortran
This generates the libraries for oed
----------Edit lked script----------------------
in lked script CHANGE THIS LINE ADDING OED LIB
if($SIMGMS == true) then
set SIMGMSLIBS = "-L$GMS_BUILD_DIR/simgms/Debug -lsimgms -L$GMS_BUILD_DIR/simgms/oed -liboed -liboedlib -lstdc++ -L$GMS_BUILD_DIR/simgms/simint/lib -lsimint"
-liboed -liboedlib was wrong
Replace by -loed -loedlib
if($SIMGMS == true) then
set SIMGMSLIBS = "-L$GMS_BUILD_DIR/simgms/Debug -lsimgms -L$GMS_BUILD_DIR/simgms/oed -loed -loedlib -lstdc++ -L$GMS_BUILD_DIR/simgms/simint/lib -lsimint"
./lked gamess 01 >& lked_may_1.log &
changed file with wrong paths that were hardcoded
Debug/simgms_diagonalize.d
---------------------may 3---
changed linked order to simgms, simint ... etc
ran lked
./lked gamess 01 >& lked_may_3.log &
openmpi path is wrong in $PATH
rhe16 doesn't exist
available directory is:
/opt/shared/sw/x86_64-unknown-linux-gnu/openmpi
Errors in lked are from intel function calls from these files:
1. shell.h , there is no shell.c
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint
change simint_config.h
remove #define SIMINT_INTEL and add #define SIMINT_GCC
now simint need to be compiled
how do we do this???
------------------------------------------------------may 5 ------------------
extracted and renamed simint version 7 to simint
created build directory in simint dir
First run cmake in the toplevel and this creates the subdirectory simint and the cmakelist there.
ran cmake command with avx
simint>> CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=avx
simgms/simint>> make
This builds all *.o objects
But its very slow.. next time try -j8. Then ran make all
[umunipala@newark simint]$ pwd
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint
[umunipala@newark simint]$ make all DESTDIR=.
------------------------
Now we have to run cmake in the subdirectories examples and then in simint.
1)
now goto /siminit/examples>> run cmake
CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=avx -DCMAKE_INSTALL_LIBDIR=./lib -DCMAKE_INSTALL_INCLUDEDIR=./include
2)
now goto /siminit/simint>> and run cmake to build the library
Also create lib and include directories here and give them as cmake variables.
[umunipala@newark simint]$ pwd
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint
[umunipala@newark simint]$ CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=avx -DCMAKE_INSTALL_LIBDIR=./lib -DCMAKE_INSTALL_INCLUDEDIR=./include
added this line to the beginning of file CMakeLists in simgms/simgms dir before running cmake :
cmake_minimum_required(VERSION 3.3)
make
make all
libsimint.a is created in simint. copy this to simint/lib
-----------------------------
goto lked and run it.
./lked gamess 01 >& lked_may_5.log &
errors libmkl_gnu_thread.so: undefined reference to `omp_in_parallel' etc
added -lgomp to mkl Lib path
linked binaries to GAMESS : sucess!!!
----------------------------
goto machines/readme.unix
changed script rungms
hostname, src path etc
then ran ./compddi
if i run the test/runtest :: has ifort errors
see the .compddi again and try to change cc etc..
------------------------------------may 8
run lked created path /tmp/gamess
changes to --rungms----
556
set DDI_MPI_CHOICE=mpich
changed setting library path etc.. in this file.
------------rungms script is optimized for intel mpi only. We need to optimize this script for gcc mpich
e.g:line 764
setenv I_MPI_WAIT_MODE enable
setenv I_MPI_PIN disable
setenv I_MPI_DEBUG 0
setenv I_MPI_STATS 0
# next two select highest speed mode of an Infiniband
setenv I_MPI_FABRICS dapl
setenv I_MPI_DAT_LIBRARY libdat2.so
These are optimizations for impi
------------------
what about MKL path??
cuda path is given in line 1117
maybe only needed if ga is set
-----------------------------------------
what are the dual gateways called?? like xcams
maybe try with eclipse?
************************************************************************************************************************************************************************************************************************************************************************************************************************************************
C CODE on Newark
-----------------jul 10--------------------------------------------
--------------------------------------------------------------------
ported modified c and c++ code.
generated oed libs
ostei_config.h is in simint/stage folder. It is not the same as config.h.in
copy it to ostei. @ is replaced with numbers in this file.
now run make in /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint
to generate simint libray.
Error generating simgms lib:
from ../simgms_init.cpp:1:
/usr/include/stdint.h:56:27: note: previous declaration as 'typedef long unsigned int uint64_t'
typedef unsigned long int uint64_t;
>simgms/debug>
g++ -Wall -mavx -g ../main.cpp ../simgms_scf__c.o simgms_init_c.o -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug/lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lsimint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main ../main.cpp ../simgms_scf__c.c ../simgms_init_c.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug/lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lsimint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include
----------------July 19:
Changed subdir.mk and added changes for c files.
created libsimgms.a from all c and cpp files except main.
----Trying to call the libraries and make an executable for main.
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include
maybe add the rest of the libraries?
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib64 -libstdc -libsupc++ -libtsan -libubsan -libvtv -libquadmath -libgcc_s -libgcj -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -liboed -liboedlib -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include
--------------7/21
fixed error from simgms_oed
compiling...............change subdir.mk
Debug>
g++ -Wall -mavx -g -O -O1 -std=c++11 -L/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib64 -libstdc -libsupc++ -libtsan -libubsan -libvtv -libquadmath -libgcc_s -libgcj -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/debug -DDEBUG -DMKL_ILP64 -MMD -MP -MF"$(@:%.o=%.d)" -MT"$(@:%.o=%.d)" -c -o "$@" "$<"
linking....................
Debug>
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include
--------------7/21
fixed error from simgms_oed
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include
main is generated: illigal instruction error: Program received signal SIGILL, Illegal instruction
libgomp would not support mkl parallel
changing linked libraries
removed lgomp and mkl_gnu_thread, still same runtime error
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -lpthread -lm -ldl
generating executable
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -lpthread
Error might be due to a different linux version
gfortran might not work directly with mkl: need to add libraries
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_gf_ilp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -lpthread
added libmkl_gnu_thread.a and libmkl_gf_ilp64.a and gomp as intel_mkl thread suggested. --> same error.
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_gf_ilp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -lpthread
1. under windows, MKL only support libiomp5 or pgi openmp Run time library, no including gfortran openmp.
2. if under Linux, mkl can support gfortran openmp. The link line is like https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
changed lmkl_gf_ilp64 into lmkl_gf_lp64
aded -lmkl_blas95_lp64 for cblas_dgem call
Trying with plain -fopenmp:
g++ -Wl,--start-group -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
----jul 25 Removing all -mavx
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
the program crashes with SIGILL on the very first AVX instruction (vmovsd). This means that your hardware, hypervisor, or kernel does not support AVX. If your hardware supports AVX, you can likely get this to work by software upgrades (hypervisor and kernel). If you use virtualization, you may have to instruct the hypervisor to pass through AVX support to the guest, too. (Recent enough kernels will automatically support AVX
Was able to get it working by the above fix.
>gdb main
>b main
>run
>when it throws illegal expression get disassembler
> disassemble /m
> change the settings to intel from att
> set disassembly-flavor intel
Dump of assembler code for function simint_normalize_shells:
0x0000000000412fb0 <+0>: test edi,edi
0x0000000000412fb2 <+2>: jle 0x413171
0x0000000000412fb8 <+8>: lea edx,[rdi-0x1]
0x0000000000412fbb <+11>: push r15
0x0000000000412fbd <+13>: push r14
0x0000000000412fbf <+15>: push r13
0x0000000000412fc1 <+17>: push r12
0x0000000000412fc3 <+19>: shl rdx,0x6
0x0000000000412fc7 <+23>: push rbp
0x0000000000412fc8 <+24>: push rbx
0x0000000000412fc9 <+25>: lea rbp,[rsi+0x4]
0x0000000000412fcd <+29>: lea r15,[rsi+rdx*1+0x44]
0x0000000000412fd2 <+34>: sub rsp,0x48
0x0000000000412fd6 <+38>: movsxd r14,DWORD PTR [rbp-0x4]
=> 0x0000000000412fda <+42>: vxorpd xmm0,xmm0,xmm0
0x0000000000412fde <+46>: vmovsd xmm6,QWORD PTR [rip+0x56052] # 0x469038
---------july 26
added -march=native to subdir.mk g++ and gcc comands. compiling same with main :
SIMINT_SCALAR is defined in vectorization.h and all others who use it inherits SIMINT_SCALAR: no errrors there.
It is defind as SIMINT_AVX in vectorization/vector_config.h
changed it to scalar and ran make to build simint library.
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
After recreating simint move to lib and also copy headers to include
--7/26 3.30 building openarc : already downloaded
Generate main from Debug:
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
. /opt/shared/sw/x86_64-unknown-linux-gnu/modules-tcl85/default/init/bash
module load modules
module use /opt/shared/sw/x86_64-unknown-linux-gnu/modulefiles/rhel6
#module use /opt/shared/sw/x86_64-unknown-linux-gnu/modulefiles/rhel7
then source or login again
. ~/.bashrc - sourcing bashrc: need to be in bash shell for this command
doesn't work for tcsh. no modules available in tcsh? why?
****If you are out of newark and load modules when u are in newark the paths are still there.Run config file in tcsh this way ; else it can't find modules
modules loaded in newark
d Modulefiles:
1) modules 3) boost/1.54.0-intel 5) python/2.7.11 7) hdf5/1.8.10-intel
2) m4/1.4.15 4) openssl/1.0.2c 6) openmpi/1.10.1-gnu
set env variables for games
export GMS_BUILD_DIR='/home/umunipala/gamess/gamess-kris-simgms
'
export GMS_TARGET=''
export GMS_PATH='/home/umunipala/gamess/gamess-kris-simgms
'
export GMS_FORTRAN='ifort'
export GMS_MATHLIB=''
set GMS_DDI_COMM=''
export GMS_TARGET=linux64
export GMS_PATH=/home/umunipala/gamess/gamess-kris-simgms
check if shell is interactive
[[ $- == *i* ]] && echo 'Interactive' || echo 'Not interactive'
----------------------------------------------4/13
Currently Loaded Modulefiles:
Currently Loaded Modulefiles: openssl is loaded by openmpi
1) modules 4) hdf5/1.8.10-intel 7) boost/1.60.0-gnu5
2) openssl/1.0.2c 5) openmpi/1.10.1-gnu 8) mkl/2013.5.192
3) python/2.7.11 6) m4/1.4.15 9) intel/2013.5.192
RUNNING CONFIG
>tcsh
>source ./config
Errors: gcc gfortran didn't show up . Logged in and out and its shown in
[umunipala@newark ~]$ which gcc
/usr/lib64/ccache/gcc
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)
Alas, your version of gfortran does not support REAL*16,
so relativistic integrals cannot use quadruple precision.
Other than this, everything will work properly.
MKL configuration
MKL version (or 'proceed')? proceed
Math library 'mkl' will be taken from /opt/intel/mkl/lib/intel64
---------------------------
Currently Loaded Modulefiles:
1) modules 6) hdf5/1.8.10-intel
2) mkl/2013.5.192 7) m4/1.4.15
3) intel/2013.5.192 8) boost/1.54.0-intel
4) openssl/1.0.2c 9) openmpi/1.10.1-gnu
5) python/2.7.11 10) mpich/3.1.2-gnu
mpich path::
set to::
/opt/shared/sw/x86_64-unknown-linux-gnu/mpich/3.1.2-gnu-rhel6
bin is here.
5) The subdirectory gamess/ddi contains a script 'compddi' to compile
the computer science support for parallel GAMESS, known as the
Distributed Data Interface (DDI).
There is one possible customization of 'compddi' that must be done
by hand-editing 'compddi', prior to running it. If your system's
number of cores per node or total number of nodes is large, reset
the defaults of MAXCPUS=8 and MAXNODES=128.
>did not change coz its only 8
>compddi : got error due to mpi
/opt/shared/sw/x86_64-unknown-linux-gnu/mpich/include/mpi.h
fixed it by changing mpi path to this(above)
compiled::
games>./compall
Thu Apr 13 15:17:10 EDT 2017
0.063u 0.018s 0:00.10 70.0% 0+0k 16+24io 0pf+0w
------------------- done with all compilations --------------------
linking failure : missing simgms.o
------------------4-17
**./comp simgms add line to compall
>./compal : no errors.
goto games/machines>
look in readme.unix step 7
goto games> run lked script to link libraries
./lked gamess 01 >& lked.log &
** change set SIMGMSLIBS line in lked script (this is not set for current user;hardcoded.)
set SIMGMSLIBS = "-L/home/kris/gamess/simgms/Debug -lsimgms -lstdc++ -L/home/kris/gamess/simgms/simint/lib -lsimint"
change the above to:
set SIMGMSLIBS = "-L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -lstdc++ -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/lib -lsimint"
/home/umunipala/gamess/gamess-kris-simgms/simgms>>>
Readme says,
Add simgms.src to gamess source directory and use modified lked to link with gamess binary
/home/umunipala/gamess/gamess-kris-simgms/simgms/gamess_files>>
there is a lked and simgms.src
lked looks like the original config script
*****************************************Above might not be necessory
move simgms to source directory.
goto:
/home/umunipala/gamess/gamess-kris-simgms/simgms/gamess_files>>
cp simgms.src ../../source/simgms.src
ran the lked in this directory: asks to run config.
replaced original lked in games>> same error
./lked gamess 01 >& lked.log &
run /.compal
_____________________________________________________4/18 changed mkl path in config
lked errors, makefile in simgms/debug had licencing errors with intel
>Going back to configuring with gcc because the makefile is autogenerated.
/opt/intel/composer_xe_2013.5.192/mkl
changed modules
in readme.unix that is in machines folder says
If you are using the Ubuntu version of Linux, you will get better results by changing the top line of each script to read
#!/bin/tcsh in all the scripts below:
config compall comp lked rungms runall
run the compddi in gamess/ddi
run compall after going back to bash and loading modules.
reverted from tcsh to bash and changed to #!/bin/csh in compall
goto games/simgms/Debug> make -icpc=cc
ran ./lked gamess 01 >& lked.log & and still there is no libsimgms.a or .so
checked with find . -name libsimgms*
not available.
module load gcc
module load make
module load python/2.7.11
module load openmpi/1.10.1-gnu
module load hdf5/1.8.10-intel
module load boost/1.60.0-gnu5
module load mkl/2013.5.192
module load mpich/3.1.2-gnu
____________________4/24/17
changing makefiles in Debug folder
subdir.mk : contains wrong path
/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 is mkl lib and they are shared/dynamic
-std=gnu++11 -lmkl_intel_ilp64 -lmkl_gnu_thread
/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
setenv LD_LIBRARY_PATH /opt/intel/composer_xe_2013.5.192/mkl/lib/intel64:$LD_LIBRARY_PATH
-----------------------------4/26---------
fix intrinsics_avx.h error
________________________-approach 1--didn't work
goto /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/vectorization
>>
current files here are:
intrinsics_avx512.h intrinsics_scalar.h vectorization.h
intrinsics_avx.h intrinsics_sse.h
>> Now adding headers..
change the header from <> to "immintrin.h" in intrinsics_avx.h
>>gcc libs
/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib/gcc/x86_64-unknown-linux-gnu/5.3.0/include
-------------------------------approach 2-----------------
define your own macro like this in intrinsics_avx.h
#define _mm256_set_m128d(vh, vl) \
_mm256_insertf128_pd(_mm256_castpd128_pd256(vl), (vh), 1)
Instead #define SIMINT_GCC in intrinsics_avx.h
------------------------------------------------------------make o/p
[umunipala@newark Debug]$ make
Building file: ../simgms_init.cpp
Invoking: G++ Compiler
g++ -g -O -O1 -std=c++11 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -libmkl_lapack95_lp64 -libmkl_intel_lp64 -libmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -DDEBUG -DMKL_ILP64 -MMD -MP -MF"simgms_init.d" -MT"simgms_init.d" -c -o "simgms_init.o" "../simgms_init.cpp"
In file included from /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/incl
ude/simint/vectorization/vectorization.h:11:0,
from /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/incl
ude/simint/ostei/ostei_config.h:3,
from /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/incl
ude/simint/simint.h:5,
from ../simgms_init.h:3,
from ../simgms_init.cpp:3:
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/vectorizati
on/intrinsics_avx.h: In function '__m256d simint_exp_vec4(__m256d)':
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/vectorizati
on/intrinsics_avx.h:25:48: warning: AVX vector return without AVX enabled changes the ABI [-Wpsabi]
static inline __m256d simint_exp_vec4(__m256d x)
^
Finished building: ../simgms_init.cpp
Building file: ../simgms_math.cpp
Invoking: G++ Compiler
g++ -g -O -O1 -std=c++11 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -libmkl_lapack95_lp64 -libmkl_intel_lp64 -libmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -DDEBUG -DMKL_ILP64 -MMD -MP -MF"simgms_math.d" -MT"simgms_math.d" -c -o "simgms_math.o" "../simgms_math.cpp"
../simgms_math.cpp:2:25: fatal error: mkl_lapacke.h: No such file or directory
compilation terminated.
---------------------------------
/opt/intel/composer_xe_2013.5.192/mkl????????????????? sometimes this path shows as available and sometimes not? why?
/opt/intel/composer_xe_2013.5.192/mkl/include
has mkl_lapack.h file
-----------------------------------------------------------
generated libsimint.a in libs and there is two more files simgms_init.d and simgms_init.o generated.
compiling lked script in main directory
./lked gamess 01 >& lked.log &
-------------------------------------------------current build error
gcc -mavx -g -O -O1 -std=c++11 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -libmkl_mc -libmkl_sequential -libmkl_gf_ilp64 -libmkl_avx -libmkl_lapack95_lp64 -libmkl_intel_lp64 -libmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -DDEBUG -DMKL_ILP64 -MMD -MP -MF"simgms_math.d" -MT"simgms_math.d" -c -o "simgms_math.o" "../simgms_math.cpp"
In file included from ../simgms_math.cpp:2:0:
../mkl_lapacke.h:32:23: fatal error: mkl_types.h: No such file or directory
compilation terminated.
subdir.mk:31: recipe for target 'simgms_math.o' failed
make: *** [simgms_math.o] Error 1
-------------------------------------
simgms/Debug>
got an error 'CBLAS_LAYOUT undeclared
fixed it with adding line typedef CBLAS_ORDER CBLAS_LAYOUT; /* this for backward compatibility with CBLAS_ORDER */
to simgms_math.cpp
Ran make in Debug folder.
This generated libsimgs.a
go up to main lked script
________________________________modified headers in simgms_math.cpp
added include files.
/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64
./lked gamess 01 >& lked_may_1.log &
libs gcc
/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib64
added this line to simint.
define
look for files in /opt/intel/composer_xe_2013.5.192/compiler/lib/intel64
gcc -I/opt/intel/composer_xe_2013.5.192/compiler/include-L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 __intel_ssse3_rep_memcpy
-----
added macro #define SIMINT_GC in simgms_init.cpp and simgms_oei.cpp --- not sure: bcoz it was mentioned in vectorization.h
---------------------may 1
copied all the header files from the Debug directory generated from the files like simgms_math, simgms_oie etc (which we in Debug )into the folder below.
Then added the bath in subdir.mk
-I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/debug
./lked gamess 01 >& lked_may_1.log &
*** Run makefile from following directory:
/home/umunipala/gamess/gamess-kris-simgms/simgms/oed
changed ifort to gfortran in Makefile
FC=gfortran
This generates the libraries for oed
----------Edit lked script----------------------
in lked script CHANGE THIS LINE ADDING OED LIB
if($SIMGMS == true) then
set SIMGMSLIBS = "-L$GMS_BUILD_DIR/simgms/Debug -lsimgms -L$GMS_BUILD_DIR/simgms/oed -liboed -liboedlib -lstdc++ -L$GMS_BUILD_DIR/simgms/simint/lib -lsimint"
-liboed -liboedlib was wrong
Replace by -loed -loedlib
if($SIMGMS == true) then
set SIMGMSLIBS = "-L$GMS_BUILD_DIR/simgms/Debug -lsimgms -L$GMS_BUILD_DIR/simgms/oed -loed -loedlib -lstdc++ -L$GMS_BUILD_DIR/simgms/simint/lib -lsimint"
./lked gamess 01 >& lked_may_1.log &
changed file with wrong paths that were hardcoded
Debug/simgms_diagonalize.d
---------------------may 3---
changed linked order to simgms, simint ... etc
ran lked
./lked gamess 01 >& lked_may_3.log &
openmpi path is wrong in $PATH
rhe16 doesn't exist
available directory is:
/opt/shared/sw/x86_64-unknown-linux-gnu/openmpi
Errors in lked are from intel function calls from these files:
1. shell.h , there is no shell.c
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint
change simint_config.h
remove #define SIMINT_INTEL and add #define SIMINT_GCC
now simint need to be compiled
how do we do this???
------------------------------------------------------may 5 ------------------
extracted and renamed simint version 7 to simint
created build directory in simint dir
First run cmake in the toplevel and this creates the subdirectory simint and the cmakelist there.
ran cmake command with avx
simint>> CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=avx
simgms/simint>> make
This builds all *.o objects
But its very slow.. next time try -j8. Then ran make all
[umunipala@newark simint]$ pwd
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint
[umunipala@newark simint]$ make all DESTDIR=.
------------------------
Now we have to run cmake in the subdirectories examples and then in simint.
1)
now goto /siminit/examples>> run cmake
CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=avx -DCMAKE_INSTALL_LIBDIR=./lib -DCMAKE_INSTALL_INCLUDEDIR=./include
2)
now goto /siminit/simint>> and run cmake to build the library
Also create lib and include directories here and give them as cmake variables.
[umunipala@newark simint]$ pwd
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint
[umunipala@newark simint]$ CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=avx -DCMAKE_INSTALL_LIBDIR=./lib -DCMAKE_INSTALL_INCLUDEDIR=./include
added this line to the beginning of file CMakeLists in simgms/simgms dir before running cmake :
cmake_minimum_required(VERSION 3.3)
make
make all
libsimint.a is created in simint. copy this to simint/lib
-----------------------------
goto lked and run it.
./lked gamess 01 >& lked_may_5.log &
errors libmkl_gnu_thread.so: undefined reference to `omp_in_parallel' etc
added -lgomp to mkl Lib path
linked binaries to GAMESS : sucess!!!
----------------------------
goto machines/readme.unix
changed script rungms
hostname, src path etc
then ran ./compddi
if i run the test/runtest :: has ifort errors
see the .compddi again and try to change cc etc..
------------------------------------may 8
run lked created path /tmp/gamess
changes to --rungms----
556
set DDI_MPI_CHOICE=mpich
changed setting library path etc.. in this file.
------------rungms script is optimized for intel mpi only. We need to optimize this script for gcc mpich
e.g:line 764
setenv I_MPI_WAIT_MODE enable
setenv I_MPI_PIN disable
setenv I_MPI_DEBUG 0
setenv I_MPI_STATS 0
# next two select highest speed mode of an Infiniband
setenv I_MPI_FABRICS dapl
setenv I_MPI_DAT_LIBRARY libdat2.so
These are optimizations for impi
------------------
what about MKL path??
cuda path is given in line 1117
maybe only needed if ga is set
-----------------------------------------
what are the dual gateways called?? like xcams
maybe try with eclipse?
************************************************************************************************************************************************************************************************************************************************************************************************************************************************
C CODE on Newark
-----------------jul 10--------------------------------------------
--------------------------------------------------------------------
ported modified c and c++ code.
generated oed libs
ostei_config.h is in simint/stage folder. It is not the same as config.h.in
copy it to ostei. @ is replaced with numbers in this file.
now run make in /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint
to generate simint libray.
Error generating simgms lib:
from ../simgms_init.cpp:1:
/usr/include/stdint.h:56:27: note: previous declaration as 'typedef long unsigned int uint64_t'
typedef unsigned long int uint64_t;
>simgms/debug>
g++ -Wall -mavx -g ../main.cpp ../simgms_scf__c.o simgms_init_c.o -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug/lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lsimint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main ../main.cpp ../simgms_scf__c.c ../simgms_init_c.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug/lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lsimint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include
----------------July 19:
Changed subdir.mk and added changes for c files.
created libsimgms.a from all c and cpp files except main.
----Trying to call the libraries and make an executable for main.
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include
maybe add the rest of the libraries?
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib64 -libstdc -libsupc++ -libtsan -libubsan -libvtv -libquadmath -libgcc_s -libgcj -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -liboed -liboedlib -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include
--------------7/21
fixed error from simgms_oed
compiling...............change subdir.mk
Debug>
g++ -Wall -mavx -g -O -O1 -std=c++11 -L/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib64 -libstdc -libsupc++ -libtsan -libubsan -libvtv -libquadmath -libgcc_s -libgcj -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/debug -DDEBUG -DMKL_ILP64 -MMD -MP -MF"$(@:%.o=%.d)" -MT"$(@:%.o=%.d)" -c -o "$@" "$<"
linking....................
Debug>
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include
--------------7/21
fixed error from simgms_oed
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include
main is generated: illigal instruction error: Program received signal SIGILL, Illegal instruction
libgomp would not support mkl parallel
changing linked libraries
removed lgomp and mkl_gnu_thread, still same runtime error
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -lpthread -lm -ldl
generating executable
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -lpthread
Error might be due to a different linux version
gfortran might not work directly with mkl: need to add libraries
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_gf_ilp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -lpthread
added libmkl_gnu_thread.a and libmkl_gf_ilp64.a and gomp as intel_mkl thread suggested. --> same error.
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_gf_ilp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -lpthread
1. under windows, MKL only support libiomp5 or pgi openmp Run time library, no including gfortran openmp.
2. if under Linux, mkl can support gfortran openmp. The link line is like https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
changed lmkl_gf_ilp64 into lmkl_gf_lp64
aded -lmkl_blas95_lp64 for cblas_dgem call
Trying with plain -fopenmp:
g++ -Wl,--start-group -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
----jul 25 Removing all -mavx
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
the program crashes with SIGILL on the very first AVX instruction (vmovsd). This means that your hardware, hypervisor, or kernel does not support AVX. If your hardware supports AVX, you can likely get this to work by software upgrades (hypervisor and kernel). If you use virtualization, you may have to instruct the hypervisor to pass through AVX support to the guest, too. (Recent enough kernels will automatically support AVX
Was able to get it working by the above fix.
>gdb main
>b main
>run
>when it throws illegal expression get disassembler
> disassemble /m
> change the settings to intel from att
> set disassembly-flavor intel
Dump of assembler code for function simint_normalize_shells:
0x0000000000412fb0 <+0>: test edi,edi
0x0000000000412fb2 <+2>: jle 0x413171
0x0000000000412fb8 <+8>: lea edx,[rdi-0x1]
0x0000000000412fbb <+11>: push r15
0x0000000000412fbd <+13>: push r14
0x0000000000412fbf <+15>: push r13
0x0000000000412fc1 <+17>: push r12
0x0000000000412fc3 <+19>: shl rdx,0x6
0x0000000000412fc7 <+23>: push rbp
0x0000000000412fc8 <+24>: push rbx
0x0000000000412fc9 <+25>: lea rbp,[rsi+0x4]
0x0000000000412fcd <+29>: lea r15,[rsi+rdx*1+0x44]
0x0000000000412fd2 <+34>: sub rsp,0x48
0x0000000000412fd6 <+38>: movsxd r14,DWORD PTR [rbp-0x4]
=> 0x0000000000412fda <+42>: vxorpd xmm0,xmm0,xmm0
0x0000000000412fde <+46>: vmovsd xmm6,QWORD PTR [rip+0x56052] # 0x469038
---------july 26
added -march=native to subdir.mk g++ and gcc comands. compiling same with main :
SIMINT_SCALAR is defined in vectorization.h and all others who use it inherits SIMINT_SCALAR: no errrors there.
It is defind as SIMINT_AVX in vectorization/vector_config.h
changed it to scalar and ran make to build simint library.
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
After recreating simint move to lib and also copy headers to include
--7/26 3.30 building openarc : already downloaded
Generate main from Debug:
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
Jul 7, 2017
Writing a makefile for a linux project
Say we have files main.c hello.c math.c and math.h
Suppose main calls hello() and then math()
gcc Steps
Preprocess
1. Removes all preprocessor directives (e.g: #define etc)
2. produces .i files from .c files
main.i hello.i
Compile
1. Converts to assembly
2. Produces main.s hello.s math.s
Assembling
1.Converts to machine language with unresolved directives
2. Produces main.o hello.o and math.o binaries
Link
gcc -o main main.o hello.o math.o -lc
1.Creates machine language executable
2.Produces main binary
Useful shortcuts
make all : creates the final binary
make clobber : delete all temp files, core files, binaries, etc.
make clean : delete all binaries
files ending ~ and starting or ending in # are temp files
core is a file produced when a program dumps core
Libraries
Libraries are files of ready-compiled code that compiler links with the c programs during linker stage.
Jun 23, 2017
openARC
No llvm support enabled.
export OPENARC_ARCH=3
export ACC_DEVICE_TYPE=Altera FPGAs
OMP_NUM_THREADS
(optional)export OPENARC_JITOPTION="-I . -I $openarc/openarcrt"
OPENARCRT_UNIFIEDMEM
OPENARCRT_VERBOSITY
export openarc=/home/umunipala/openARC
$ [openarc-path]/bin/openarc [options] [C files]
$ /home/umunipala/openARC/bin/openarc [options] matmul.c
export OPENARC_ARCH=3
export ACC_DEVICE_TYPE=Altera FPGAs
OMP_NUM_THREADS
(optional)export OPENARC_JITOPTION="-I . -I $openarc/openarcrt"
OPENARCRT_UNIFIEDMEM
OPENARCRT_VERBOSITY
export openarc=/home/umunipala/openARC
$ [openarc-path]/bin/openarc [options] [C files]
$ /home/umunipala/openARC/bin/openarc [options] matmul.c
May 27, 2017
simint/test cmake build
source modules in newark,
load module cmake
mkdir build
cd build
cmake ..
CC=gcc CXX=g++ CMAKE_C_COMPILER=gcc CMAKE_CXX_COMPILER=g++ cmake .. -DSIMINT_VECTOR=avx
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:usr/lib64
cd build
std=gnu++11 CMAKE_SOURCE_DIR=/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/example cmake ..
load module cmake
mkdir build
cd build
cmake ..
CC=gcc CXX=g++ CMAKE_C_COMPILER=gcc CMAKE_CXX_COMPILER=g++ cmake .. -DSIMINT_VECTOR=avx
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:usr/lib64
cd build
std=gnu++11 CMAKE_SOURCE_DIR=/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/example cmake ..
cd ..
-------------------------------------
[umunipala@newark examples]$ make
[ 25%] Linking C executable example1
/usr/local/bin/ld: cannot find -lsimint
collect2: error: ld returned 1 exit status
CMakeFiles/example1.dir/build.make:94: recipe for target 'example1' failed
make[2]: *** [example1] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/example1.dir/all' failed
make[1]: *** [CMakeFiles/example1.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
[umunipala@newark examples]$ make clean
[umunipala@newark examples]$ make
[ 25%] Building C object CMakeFiles/example1.dir/example1.c.o
[ 50%] Linking C executable example1
/usr/local/bin/ld: cannot find -lsimint
collect2: error: ld returned 1 exit status
CMakeFiles/example1.dir/build.make:94: recipe for target 'example1' failed
make[2]: *** [example1] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/example1.dir/all' failed
make[1]: *** [CMakeFiles/example1.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
[umunipala@newark examples]$ vi CMakeLists.txt
cmake_minimum_required(VERSION 3.3)
-----------------------------------------------------------------------------------------------------
copied simgms original version
check for include errors
error: /usr/local/bin/ld: cannot find -lsimint
How do we add the externel lib?
changing cmakelists.txt didn't work
target_compile_options
link_directories(
This seems to be the solution: didn't work
-----------------------------------------------------------------------------------------------------
copied simgms original version
check for include errors
error: /usr/local/bin/ld: cannot find -lsimint
How do we add the externel lib?
changing cmakelists.txt didn't work
target_compile_options
link_directories(
This seems to be the solution: didn't work
-------------------------------------------------------------------------------------------------
#include simint/ostei/ostei.h is missing in vb and original versio??
#include simint/ostei/ostei.h is missing in vb and original versio??
May 26, 2017
simgms Visual Studio Built
globaly define SIMINT_GCC
Vectorization.h
#define SIMINT_SSE
ostei_config.h
#define SIMINT_OSTEI_MAXAM 0
#define SIMINT_OSTEI_MAXDER 0
shell.h
//#define SIMINT_OSTEI_MAXDER @SIMINT_MAXDER@
#define SIMINT_MAXDER 0
simgms_tei.cpp
#define SIMINT_OSTEI_MAX_WORKMEM 0
simgms_oei.cpp
changed the arrays CC and ALPHA to work with visual studio : previous was ok for gcc only.
added mkl headers to path
simgms_math
MKL_INT *ipiv;
ipiv = (MKL_INT *)malloc(n * sizeof(MKL_INT));
shell.c
P->x = (double*)P->ptr + dprim_size*(dcount++);
shell_screen.c
///double integrals[ncart1234] SIMINT_ALIGN_ARRAY_DBL;
double *integrals;
integrals = (double *)aligned_alloc(SIMINT_SIMD_ALIGN_DBL, ncart1234 * sizeof(double));
Vectorization.h
#define SIMINT_SSE
ostei_config.h
#define SIMINT_OSTEI_MAXAM 0
#define SIMINT_OSTEI_MAXDER 0
shell.h
//#define SIMINT_OSTEI_MAXDER @SIMINT_MAXDER@
#define SIMINT_MAXDER 0
simgms_tei.cpp
#define SIMINT_OSTEI_MAX_WORKMEM 0
simgms_oei.cpp
changed the arrays CC and ALPHA to work with visual studio : previous was ok for gcc only.
added mkl headers to path
simgms_math
MKL_INT *ipiv;
ipiv = (MKL_INT *)malloc(n * sizeof(MKL_INT));
shell.c
P->x = (double*)P->ptr + dprim_size*(dcount++);
shell_screen.c
///double integrals[ncart1234] SIMINT_ALIGN_ARRAY_DBL;
double *integrals;
integrals = (double *)aligned_alloc(SIMINT_SIMD_ALIGN_DBL, ncart1234 * sizeof(double));
2 locations
May 11, 2017
Build / Debug c/c++ synchronized projects behind ssh remote gateway via Eclipse ( Client-Server type)
I found tunneling a remote gateway very slow and figured out a better way to work with synchronized projects. {This is very easy via ssh with -xy enabled on server and tunnel through x-window system (Xming For windows)}.
Login as
ssh -XY username@xx.yy.org in both the gateway and the machine.
I couldn't find a way to use Netbeans for remote C/C++/Fortran projects with a gateway yet. Below method is using Eclipse for linux or windows.
Files you have edit
Remote:
Your bashrc script or bash_profile - This is hidden in /home/username
Do : ll -a then vi bash_profile
Remote development by synchronized projects uses git. You need git installed on your remote machine. So we need to add the lines below to the end of your bash_profile script.
Local:
Eclipse seem to support Git inherently. So we only need to install eclipse neon for parallel C/C++ development
Eclipse settings --------------------------------------------------------------------------------
Goto Window>Preferences>Add
Then click open button on the right hand side. This will open up the connection. If you didn't save your password, you have to enter the password now.
Next go to menu File>New>Other from the menu.
Select the project type as GNU Auto Tools (This could be different depending on your project.) and select 'Empty project'.
credit: https://www.earthsystemcog.org/projects/cupid/proxyconnections and the great WWW!
******************************************************************************
More on setting up include paths for the remote synchronized project to run using remote machine:
(If needed, Click the link to goto eclipse help location which I'm quoting below )
"There are several places where include files are important to the editing, analyzing, building, and running of your program. A Synchronized Project mirrors the source files on the local machine as well as the remote target machine. Usually you edit locally, but build and run on the remote target. So this complicates things with respect to include files.
Now, make sure that indexing is enabled and invoke a remote build. In order for the build output parser to be useful, builds must produce verbose output. The method for setting up such a build varies per project, but see the Mozilla Project CDT setup page for a good example. Repeat these steps for each remote build configuration. For local configurations, both the standard CDT or new Sync providers should work. If you would like the environment to update automatically when build configurations are switched, go to Project > Properties > C/C++ General > Indexer and select "Use active build configuration" for "Build configuration for the indexer." The downside of this is that the indexer will run each time you switch configurations."
E.g: See below. Instead of using //nxxx.x.org/opt/../include use //ConnectionName/opt/../include when you set include files from remote location.
Login as
ssh -XY username@xx.yy.org in both the gateway and the machine.
I couldn't find a way to use Netbeans for remote C/C++/Fortran projects with a gateway yet. Below method is using Eclipse for linux or windows.
Files you have edit
Remote:
Your bashrc script or bash_profile - This is hidden in /home/username
Do : ll -a then vi bash_profile
Remote development by synchronized projects uses git. You need git installed on your remote machine. So we need to add the lines below to the end of your bash_profile script.
module load gitgit config --global pack.windowMemory "100m"
git config --global pack.SizeLimit "100m"git config --global pack.threads "1"Local:
Eclipse seem to support Git inherently. So we only need to install eclipse neon for parallel C/C++ development
Eclipse settings --------------------------------------------------------------------------------
Goto Window>Preferences>Add
Add your gateway as the first connection. Give the Host name and your username. ( equivalent to , ssh username@host.org etc). Connection name could be anything you want. Here I set it as 'xcams gateway'.Choose password based connection, if you want to use a password. Then click finish.
Next we add the actual connection that is behind the gateway. Click Add a new connection again.
Give the connection name , host and username as before. Then click Advanced. In SSH Proxy settings, click Remote, and select the Gateway we created before. Now click finish.
Next go to menu File>New>Other from the menu.
Then select Synchronized C/C++ projects from 'Other'.
Click Next. Then select the ssh connection (not the gateway).
Then select the Remote directory by clicking 'Browse' button.
Then Click the project folder you want to synchronize with.
Click ok. Give the Project name the same name as the project you selected.
Select the project type as GNU Auto Tools (This could be different depending on your project.) and select 'Empty project'.
**I had to do little bit more settings on windows to work because we need to support cross compilation with windows and our Linux based remote system. But it just works if you are using a linux machine.
Now choose to synchronize the project by right clicking on it unless auto sync is on by default. Screen shot is a windows system. But on Linux you need to specifically set it to sync. This may take a while if you have a rather large system on the remote machine.
** On windows 10 (This should be available on most windows OS)
goto control panel> Turn windows features on >
Click the check box saying, Windows subsystem for Linux
{This step is not necessary on Ubuntu}
credit: https://www.earthsystemcog.org/projects/cupid/proxyconnections and the great WWW!
******************************************************************************
More on setting up include paths for the remote synchronized project to run using remote machine:
(If needed, Click the link to goto eclipse help location which I'm quoting below )
"There are several places where include files are important to the editing, analyzing, building, and running of your program. A Synchronized Project mirrors the source files on the local machine as well as the remote target machine. Usually you edit locally, but build and run on the remote target. So this complicates things with respect to include files.
By default, a synchronized project uses the normal CDT discovery mechanism, which reads the local environment. To configure the new discovery elements, follow these instructions after creating a new synchronized project. Go to Project > Properties > C/C++ General > Preprocessor Include Paths, Macros etc. Make sure the remote configuration is selected at the top of the dialog. Under the Providers tab, deselect all CDT providers, except for "CDT User Setting Entries" if you would like to enter your own paths. Next, select the two Sync providers, "Sync GCC Build Output Parser" and "Sync GCC Builtin Compiler Settings." The latter provider runs a specially-crafted compiler command to get compiler-specific settings. It will run immediately after "OK" is selected. The former provider will execute simultaneously with builds. It intercepts the build output and attempts to find environment information. Thus, a build must be run before this provider is useful.
Here if you are connecting through a gateway remember to use the name of the ssh connection (2nd) instead of the full path name. Otherwise Eclipse tries to
login to the machine directly.E.g: See below. Instead of using //nxxx.x.org/opt/../include use //ConnectionName/opt/../include when you set include files from remote location.
Subscribe to:
Posts (Atom)