. /opt/shared/sw/x86_64-unknown-linux-gnu/modules-tcl85/default/init/bash
module load modules
module use /opt/shared/sw/x86_64-unknown-linux-gnu/modulefiles/rhel6
#module use /opt/shared/sw/x86_64-unknown-linux-gnu/modulefiles/rhel7
then source or login again
. ~/.bashrc - sourcing bashrc: need to be in bash shell for this command
doesn't work for tcsh. no modules available in tcsh? why?
****If you are out of newark and load modules when u are in newark the paths are still there.Run config file in tcsh this way ; else it can't find modules
modules loaded in newark
d Modulefiles:
1) modules 3) boost/1.54.0-intel 5) python/2.7.11 7) hdf5/1.8.10-intel
2) m4/1.4.15 4) openssl/1.0.2c 6) openmpi/1.10.1-gnu
set env variables for games
export GMS_BUILD_DIR='/home/umunipala/gamess/gamess-kris-simgms
'
export GMS_TARGET=''
export GMS_PATH='/home/umunipala/gamess/gamess-kris-simgms
'
export GMS_FORTRAN='ifort'
export GMS_MATHLIB=''
set GMS_DDI_COMM=''
export GMS_TARGET=linux64
export GMS_PATH=/home/umunipala/gamess/gamess-kris-simgms
check if shell is interactive
[[ $- == *i* ]] && echo 'Interactive' || echo 'Not interactive'
----------------------------------------------4/13
Currently Loaded Modulefiles:
Currently Loaded Modulefiles: openssl is loaded by openmpi
1) modules 4) hdf5/1.8.10-intel 7) boost/1.60.0-gnu5
2) openssl/1.0.2c 5) openmpi/1.10.1-gnu 8) mkl/2013.5.192
3) python/2.7.11 6) m4/1.4.15 9) intel/2013.5.192
RUNNING CONFIG
>tcsh
>source ./config
Errors: gcc gfortran didn't show up . Logged in and out and its shown in
[umunipala@newark ~]$ which gcc
/usr/lib64/ccache/gcc
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)
Alas, your version of gfortran does not support REAL*16,
so relativistic integrals cannot use quadruple precision.
Other than this, everything will work properly.
MKL configuration
MKL version (or 'proceed')? proceed
Math library 'mkl' will be taken from /opt/intel/mkl/lib/intel64
---------------------------
Currently Loaded Modulefiles:
1) modules 6) hdf5/1.8.10-intel
2) mkl/2013.5.192 7) m4/1.4.15
3) intel/2013.5.192 8) boost/1.54.0-intel
4) openssl/1.0.2c 9) openmpi/1.10.1-gnu
5) python/2.7.11 10) mpich/3.1.2-gnu
mpich path::
set to::
/opt/shared/sw/x86_64-unknown-linux-gnu/mpich/3.1.2-gnu-rhel6
bin is here.
5) The subdirectory gamess/ddi contains a script 'compddi' to compile
the computer science support for parallel GAMESS, known as the
Distributed Data Interface (DDI).
There is one possible customization of 'compddi' that must be done
by hand-editing 'compddi', prior to running it. If your system's
number of cores per node or total number of nodes is large, reset
the defaults of MAXCPUS=8 and MAXNODES=128.
>did not change coz its only 8
>compddi : got error due to mpi
/opt/shared/sw/x86_64-unknown-linux-gnu/mpich/include/mpi.h
fixed it by changing mpi path to this(above)
compiled::
games>./compall
Thu Apr 13 15:17:10 EDT 2017
0.063u 0.018s 0:00.10 70.0% 0+0k 16+24io 0pf+0w
------------------- done with all compilations --------------------
linking failure : missing simgms.o
------------------4-17
**./comp simgms add line to compall
>./compal : no errors.
goto games/machines>
look in readme.unix step 7
goto games> run lked script to link libraries
./lked gamess 01 >& lked.log &
** change set SIMGMSLIBS line in lked script (this is not set for current user;hardcoded.)
set SIMGMSLIBS = "-L/home/kris/gamess/simgms/Debug -lsimgms -lstdc++ -L/home/kris/gamess/simgms/simint/lib -lsimint"
change the above to:
set SIMGMSLIBS = "-L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -lstdc++ -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/lib -lsimint"
/home/umunipala/gamess/gamess-kris-simgms/simgms>>>
Readme says,
Add simgms.src to gamess source directory and use modified lked to link with gamess binary
/home/umunipala/gamess/gamess-kris-simgms/simgms/gamess_files>>
there is a lked and simgms.src
lked looks like the original config script
*****************************************Above might not be necessory
move simgms to source directory.
goto:
/home/umunipala/gamess/gamess-kris-simgms/simgms/gamess_files>>
cp simgms.src ../../source/simgms.src
ran the lked in this directory: asks to run config.
replaced original lked in games>> same error
./lked gamess 01 >& lked.log &
run /.compal
_____________________________________________________4/18 changed mkl path in config
lked errors, makefile in simgms/debug had licencing errors with intel
>Going back to configuring with gcc because the makefile is autogenerated.
/opt/intel/composer_xe_2013.5.192/mkl
changed modules
in readme.unix that is in machines folder says
If you are using the Ubuntu version of Linux, you will get better results by changing the top line of each script to read
#!/bin/tcsh in all the scripts below:
config compall comp lked rungms runall
run the compddi in gamess/ddi
run compall after going back to bash and loading modules.
reverted from tcsh to bash and changed to #!/bin/csh in compall
goto games/simgms/Debug> make -icpc=cc
ran ./lked gamess 01 >& lked.log & and still there is no libsimgms.a or .so
checked with find . -name libsimgms*
not available.
module load gcc
module load make
module load python/2.7.11
module load openmpi/1.10.1-gnu
module load hdf5/1.8.10-intel
module load boost/1.60.0-gnu5
module load mkl/2013.5.192
module load mpich/3.1.2-gnu
____________________4/24/17
changing makefiles in Debug folder
subdir.mk : contains wrong path
/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 is mkl lib and they are shared/dynamic
-std=gnu++11 -lmkl_intel_ilp64 -lmkl_gnu_thread
/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64
setenv LD_LIBRARY_PATH /opt/intel/composer_xe_2013.5.192/mkl/lib/intel64:$LD_LIBRARY_PATH
-----------------------------4/26---------
fix intrinsics_avx.h error
________________________-approach 1--didn't work
goto /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/vectorization
>>
current files here are:
intrinsics_avx512.h intrinsics_scalar.h vectorization.h
intrinsics_avx.h intrinsics_sse.h
>> Now adding headers..
change the header from <> to "immintrin.h" in intrinsics_avx.h
>>gcc libs
/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib/gcc/x86_64-unknown-linux-gnu/5.3.0/include
-------------------------------approach 2-----------------
define your own macro like this in intrinsics_avx.h
#define _mm256_set_m128d(vh, vl) \
_mm256_insertf128_pd(_mm256_castpd128_pd256(vl), (vh), 1)
Instead #define SIMINT_GCC in intrinsics_avx.h
------------------------------------------------------------make o/p
[umunipala@newark Debug]$ make
Building file: ../simgms_init.cpp
Invoking: G++ Compiler
g++ -g -O -O1 -std=c++11 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -libmkl_lapack95_lp64 -libmkl_intel_lp64 -libmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -DDEBUG -DMKL_ILP64 -MMD -MP -MF"simgms_init.d" -MT"simgms_init.d" -c -o "simgms_init.o" "../simgms_init.cpp"
In file included from /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/incl
ude/simint/vectorization/vectorization.h:11:0,
from /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/incl
ude/simint/ostei/ostei_config.h:3,
from /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/incl
ude/simint/simint.h:5,
from ../simgms_init.h:3,
from ../simgms_init.cpp:3:
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/vectorizati
on/intrinsics_avx.h: In function '__m256d simint_exp_vec4(__m256d)':
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/vectorizati
on/intrinsics_avx.h:25:48: warning: AVX vector return without AVX enabled changes the ABI [-Wpsabi]
static inline __m256d simint_exp_vec4(__m256d x)
^
Finished building: ../simgms_init.cpp
Building file: ../simgms_math.cpp
Invoking: G++ Compiler
g++ -g -O -O1 -std=c++11 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -libmkl_lapack95_lp64 -libmkl_intel_lp64 -libmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -DDEBUG -DMKL_ILP64 -MMD -MP -MF"simgms_math.d" -MT"simgms_math.d" -c -o "simgms_math.o" "../simgms_math.cpp"
../simgms_math.cpp:2:25: fatal error: mkl_lapacke.h: No such file or directory
compilation terminated.
---------------------------------
/opt/intel/composer_xe_2013.5.192/mkl????????????????? sometimes this path shows as available and sometimes not? why?
/opt/intel/composer_xe_2013.5.192/mkl/include
has mkl_lapack.h file
-----------------------------------------------------------
generated libsimint.a in libs and there is two more files simgms_init.d and simgms_init.o generated.
compiling lked script in main directory
./lked gamess 01 >& lked.log &
-------------------------------------------------current build error
gcc -mavx -g -O -O1 -std=c++11 -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -libmkl_mc -libmkl_sequential -libmkl_gf_ilp64 -libmkl_avx -libmkl_lapack95_lp64 -libmkl_intel_lp64 -libmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -DDEBUG -DMKL_ILP64 -MMD -MP -MF"simgms_math.d" -MT"simgms_math.d" -c -o "simgms_math.o" "../simgms_math.cpp"
In file included from ../simgms_math.cpp:2:0:
../mkl_lapacke.h:32:23: fatal error: mkl_types.h: No such file or directory
compilation terminated.
subdir.mk:31: recipe for target 'simgms_math.o' failed
make: *** [simgms_math.o] Error 1
-------------------------------------
simgms/Debug>
got an error 'CBLAS_LAYOUT undeclared
fixed it with adding line typedef CBLAS_ORDER CBLAS_LAYOUT; /* this for backward compatibility with CBLAS_ORDER */
to simgms_math.cpp
Ran make in Debug folder.
This generated libsimgs.a
go up to main lked script
________________________________modified headers in simgms_math.cpp
added include files.
/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64
./lked gamess 01 >& lked_may_1.log &
libs gcc
/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib64
added this line to simint.
define
look for files in /opt/intel/composer_xe_2013.5.192/compiler/lib/intel64
gcc -I/opt/intel/composer_xe_2013.5.192/compiler/include-L/opt/intel/composer_xe_2013.5.192/compiler/lib/intel64 __intel_ssse3_rep_memcpy
-----
added macro #define SIMINT_GC in simgms_init.cpp and simgms_oei.cpp --- not sure: bcoz it was mentioned in vectorization.h
---------------------may 1
copied all the header files from the Debug directory generated from the files like simgms_math, simgms_oie etc (which we in Debug )into the folder below.
Then added the bath in subdir.mk
-I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/debug
./lked gamess 01 >& lked_may_1.log &
*** Run makefile from following directory:
/home/umunipala/gamess/gamess-kris-simgms/simgms/oed
changed ifort to gfortran in Makefile
FC=gfortran
This generates the libraries for oed
----------Edit lked script----------------------
in lked script CHANGE THIS LINE ADDING OED LIB
if($SIMGMS == true) then
set SIMGMSLIBS = "-L$GMS_BUILD_DIR/simgms/Debug -lsimgms -L$GMS_BUILD_DIR/simgms/oed -liboed -liboedlib -lstdc++ -L$GMS_BUILD_DIR/simgms/simint/lib -lsimint"
-liboed -liboedlib was wrong
Replace by -loed -loedlib
if($SIMGMS == true) then
set SIMGMSLIBS = "-L$GMS_BUILD_DIR/simgms/Debug -lsimgms -L$GMS_BUILD_DIR/simgms/oed -loed -loedlib -lstdc++ -L$GMS_BUILD_DIR/simgms/simint/lib -lsimint"
./lked gamess 01 >& lked_may_1.log &
changed file with wrong paths that were hardcoded
Debug/simgms_diagonalize.d
---------------------may 3---
changed linked order to simgms, simint ... etc
ran lked
./lked gamess 01 >& lked_may_3.log &
openmpi path is wrong in $PATH
rhe16 doesn't exist
available directory is:
/opt/shared/sw/x86_64-unknown-linux-gnu/openmpi
Errors in lked are from intel function calls from these files:
1. shell.h , there is no shell.c
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint
change simint_config.h
remove #define SIMINT_INTEL and add #define SIMINT_GCC
now simint need to be compiled
how do we do this???
------------------------------------------------------may 5 ------------------
extracted and renamed simint version 7 to simint
created build directory in simint dir
First run cmake in the toplevel and this creates the subdirectory simint and the cmakelist there.
ran cmake command with avx
simint>> CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=avx
simgms/simint>> make
This builds all *.o objects
But its very slow.. next time try -j8. Then ran make all
[umunipala@newark simint]$ pwd
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint
[umunipala@newark simint]$ make all DESTDIR=.
------------------------
Now we have to run cmake in the subdirectories examples and then in simint.
1)
now goto /siminit/examples>> run cmake
CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=avx -DCMAKE_INSTALL_LIBDIR=./lib -DCMAKE_INSTALL_INCLUDEDIR=./include
2)
now goto /siminit/simint>> and run cmake to build the library
Also create lib and include directories here and give them as cmake variables.
[umunipala@newark simint]$ pwd
/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint
[umunipala@newark simint]$ CC=gcc CXX=g++ cmake . -DSIMINT_VECTOR=avx -DCMAKE_INSTALL_LIBDIR=./lib -DCMAKE_INSTALL_INCLUDEDIR=./include
added this line to the beginning of file CMakeLists in simgms/simgms dir before running cmake :
cmake_minimum_required(VERSION 3.3)
make
make all
libsimint.a is created in simint. copy this to simint/lib
-----------------------------
goto lked and run it.
./lked gamess 01 >& lked_may_5.log &
errors libmkl_gnu_thread.so: undefined reference to `omp_in_parallel' etc
added -lgomp to mkl Lib path
linked binaries to GAMESS : sucess!!!
----------------------------
goto machines/readme.unix
changed script rungms
hostname, src path etc
then ran ./compddi
if i run the test/runtest :: has ifort errors
see the .compddi again and try to change cc etc..
------------------------------------may 8
run lked created path /tmp/gamess
changes to --rungms----
556
set DDI_MPI_CHOICE=mpich
changed setting library path etc.. in this file.
------------rungms script is optimized for intel mpi only. We need to optimize this script for gcc mpich
e.g:line 764
setenv I_MPI_WAIT_MODE enable
setenv I_MPI_PIN disable
setenv I_MPI_DEBUG 0
setenv I_MPI_STATS 0
# next two select highest speed mode of an Infiniband
setenv I_MPI_FABRICS dapl
setenv I_MPI_DAT_LIBRARY libdat2.so
These are optimizations for impi
------------------
what about MKL path??
cuda path is given in line 1117
maybe only needed if ga is set
-----------------------------------------
what are the dual gateways called?? like xcams
maybe try with eclipse?
************************************************************************************************************************************************************************************************************************************************************************************************************************************************
C CODE on Newark
-----------------jul 10--------------------------------------------
--------------------------------------------------------------------
ported modified c and c++ code.
generated oed libs
ostei_config.h is in simint/stage folder. It is not the same as config.h.in
copy it to ostei. @ is replaced with numbers in this file.
now run make in /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint
to generate simint libray.
Error generating simgms lib:
from ../simgms_init.cpp:1:
/usr/include/stdint.h:56:27: note: previous declaration as 'typedef long unsigned int uint64_t'
typedef unsigned long int uint64_t;
>simgms/debug>
g++ -Wall -mavx -g ../main.cpp ../simgms_scf__c.o simgms_init_c.o -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug/lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lsimint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main ../main.cpp ../simgms_scf__c.c ../simgms_init_c.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug/lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lsimint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include
----------------July 19:
Changed subdir.mk and added changes for c files.
created libsimgms.a from all c and cpp files except main.
----Trying to call the libraries and make an executable for main.
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include
maybe add the rest of the libraries?
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib64 -libstdc -libsupc++ -libtsan -libubsan -libvtv -libquadmath -libgcc_s -libgcj -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -liboed -liboedlib -I /home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I /home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include
--------------7/21
fixed error from simgms_oed
compiling...............change subdir.mk
Debug>
g++ -Wall -mavx -g -O -O1 -std=c++11 -L/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/lib64 -libstdc -libsupc++ -libtsan -libubsan -libvtv -libquadmath -libgcc_s -libgcj -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/include/simint/debug -DDEBUG -DMKL_ILP64 -MMD -MP -MF"$(@:%.o=%.d)" -MT"$(@:%.o=%.d)" -c -o "$@" "$<"
linking....................
Debug>
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include
--------------7/21
fixed error from simgms_oed
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include
main is generated: illigal instruction error: Program received signal SIGILL, Illegal instruction
libgomp would not support mkl parallel
changing linked libraries
removed lgomp and mkl_gnu_thread, still same runtime error
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -lpthread -lm -ldl
generating executable
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -lpthread
Error might be due to a different linux version
gfortran might not work directly with mkl: need to add libraries
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_gf_ilp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -I/opt/shared/sw/x86_64-unknown-linux-gnu/gcc/5.3.0-rhel6/include -lpthread
added libmkl_gnu_thread.a and libmkl_gf_ilp64.a and gomp as intel_mkl thread suggested. --> same error.
g++ -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_ilp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_gf_ilp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -lpthread
1. under windows, MKL only support libiomp5 or pgi openmp Run time library, no including gfortran openmp.
2. if under Linux, mkl can support gfortran openmp. The link line is like https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
changed lmkl_gf_ilp64 into lmkl_gf_lp64
aded -lmkl_blas95_lp64 for cblas_dgem call
Trying with plain -fopenmp:
g++ -Wl,--start-group -Wall -mavx -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
----jul 25 Removing all -mavx
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
the program crashes with SIGILL on the very first AVX instruction (vmovsd). This means that your hardware, hypervisor, or kernel does not support AVX. If your hardware supports AVX, you can likely get this to work by software upgrades (hypervisor and kernel). If you use virtualization, you may have to instruct the hypervisor to pass through AVX support to the guest, too. (Recent enough kernels will automatically support AVX
Was able to get it working by the above fix.
>gdb main
>b main
>run
>when it throws illegal expression get disassembler
> disassemble /m
> change the settings to intel from att
> set disassembly-flavor intel
Dump of assembler code for function simint_normalize_shells:
0x0000000000412fb0 <+0>: test edi,edi
0x0000000000412fb2 <+2>: jle 0x413171
0x0000000000412fb8 <+8>: lea edx,[rdi-0x1]
0x0000000000412fbb <+11>: push r15
0x0000000000412fbd <+13>: push r14
0x0000000000412fbf <+15>: push r13
0x0000000000412fc1 <+17>: push r12
0x0000000000412fc3 <+19>: shl rdx,0x6
0x0000000000412fc7 <+23>: push rbp
0x0000000000412fc8 <+24>: push rbx
0x0000000000412fc9 <+25>: lea rbp,[rsi+0x4]
0x0000000000412fcd <+29>: lea r15,[rsi+rdx*1+0x44]
0x0000000000412fd2 <+34>: sub rsp,0x48
0x0000000000412fd6 <+38>: movsxd r14,DWORD PTR [rbp-0x4]
=> 0x0000000000412fda <+42>: vxorpd xmm0,xmm0,xmm0
0x0000000000412fde <+46>: vmovsd xmm6,QWORD PTR [rip+0x56052] # 0x469038
---------july 26
added -march=native to subdir.mk g++ and gcc comands. compiling same with main :
SIMINT_SCALAR is defined in vectorization.h and all others who use it inherits SIMINT_SCALAR: no errrors there.
It is defind as SIMINT_AVX in vectorization/vector_config.h
changed it to scalar and ran make to build simint library.
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_avx -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
After recreating simint move to lib and also copy headers to include
--7/26 3.30 building openarc : already downloaded
Generate main from Debug:
g++ -Wl,--start-group -Wall -g -O -O1 -std=c++11 -o main -lgfortran -fopenmp -lgomp ../main.cpp -L/home/umunipala/gamess/gamess-kris-simgms/simgms/Debug -lsimgms -L/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/lib -lsimint -L/home/umunipala/gamess/gamess-kris-simgms/simgms/oed -loed -loedlib -L/opt/intel/composer_xe_2013.5.192/mkl/lib/intel64 -lmkl_mc -lmkl_sequential -lmkl_gf_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blas95_lp64 -I/home/umunipala/gamess/gamess-kris-simgms/simgms/simint/simint/include -I/home/umunipala/gamess/gamess-kris-simgms/simgms/include -I/opt/intel/composer_xe_2013.5.192/mkl/include -Wl,--end-group -ldl -lpthread -lm
No comments:
Post a Comment