## How to Debug the Dreamview Start Problem ### Steps to Start Dreamview If you encounter problems when starting Dreamview in the `docker/scripts/dev` sequence, first check if you are using the correct commands as shown below. ```bash $ bash docker/scripts/dev_start.sh $ bash docker/scripts/dev_into.sh $ bash apollo.sh build $ bash scripts/bootstrap.sh ``` ### Dreamview Fails to Start If Dreamview fails to start, use the script below to check the Dreamview startup log and restart Dreamview. ```bash # Start Dreamview in foreground to see any error message it prints out during startup $ bash scripts/dreamview.sh start_fe # check dreamview startup log $ cat data/log/dreamview.out terminate called after throwing an instance of 'CivetException' what(): null context when constructing CivetServer. Possible problem binding to port. $ sudo apt-get install psmisc # to check if dreamview is running from other terminal $ sudo lsof -i :8888 # kill other running/pending dreamview $ sudo fuser -k 8888/tcp # restart dreamview again $ bash scripts/dreamview.sh ``` ### Debug dreamview with gdb If you get nothing in dreamview startup logs, you can try to debug dreamview with gdb, use the following commands: ``` $ gdb --args /apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/dreamview/conf/dreamview.conf # or $ source scripts/apollo_base.sh; $ start_gdb dreamview ``` Once gdb is launched, press `r` and `enter` key to run, if dreamview crashes, then get the backtrace with `bt`. ### CPU does not support FMA/FMA3 instructions If you see an error `Illegal instruction` and something related with **libpcl_sample_consensus.so.1.7** in gdb backtrace, then you probably need to rebuild pcl lib from source by yourself and replace the one in the docker. This usually happens when you're trying to run Apollo/dreamview on a machine that the CPU does not support FMA/FMA3 instructions, it will fail because the prebuilt pcl lib shipped with docker image is compiled with FMA/FMA3 support. There are 2 steps to resolve this issue: 1. Identify if the issue is due to pcl lib through gdb: find the coredump file under /apollo/data/core/ with name core_dreamview.$PID. If you see logs like: ``` @in_dev_docker:/apollo$ gdb bazel-bin/modules/dreamview/dreamview data/core/core_dreamview.378 GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from bazel-bin/modules/dreamview/dreamview...done. warning: exec file is newer than core file. [New LWP 378] [New LWP 379] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". warning: the debug information found in "/home/caros/secure_upgrade/depend_lib/libyaml-cpp.so.0.5.1" does not match "/home/caros/secure_upgrade/depend_lib/libyaml-cpp.so.0.5" (CRC mismatch). Core was generated by `/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/dreamv'. Program terminated with signal SIGILL, Illegal instruction. #0 0x00007f79ebd32bec in double boost::math::detail::erf_inv_imp, boost::math::policies::promote_double, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> >(double const&, double const&, boost::math::policies::policy, boost::math::policies::promote_double, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> const&, mpl_::int_<64> const*) () from /usr/local/lib/libpcl_sample_consensus.so.1.7 (gdb) bt #0 0x00007f79ebd32bec in double boost::math::detail::erf_inv_imp, boost::math::policies::promote_double, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> >(double const&, double const&, boost::math::policies::policy, boost::math::policies::promote_double, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> const&, mpl_::int_<64> const*) () from /usr/local/lib/libpcl_sample_consensus.so.1.7 #1 0x00007f79ebcf7f1e in _GLOBAL__sub_I_sac.cpp () from /usr/local/lib/libpcl_sample_consensus.so.1.7 #2 0x00007f79f3e1a2da in call_init (l=, argc=argc@entry=2, argv=argv@entry=0x7ffc7bc11c78, env=env@entry=0x7ffc7bc11c90) at dl-init.c:78 #3 0x00007f79f3e1a3c3 in call_init (env=, argv=, argc=, l=) at dl-init.c:36 #4 _dl_init (main_map=0x7f79f402e1c8, argc=2, argv=0x7ffc7bc11c78, env=0x7ffc7bc11c90) at dl-init.c:126 #5 0x00007f79f3e0b29a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2 #6 0x0000000000000002 in ?? () #7 0x00007ffc7bc134bd in ?? () #8 0x00007ffc7bc134eb in ?? () #9 0x0000000000000000 in ?? () (gdb) q @in_dev_docker:/apollo$ addr2line -C -f -e /usr/local/lib/libpcl_sample_consensus.so.1.7.2 0x375bec double boost::math::detail::erf_inv_imp, boost::math::policies::promote_double, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> >(double const&, double const&, boost::math::policies::policy, boost::math::policies::promote_double, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> const&, mpl_::int_<64> const*) ??:? ``` More info: ``` ~/playground/apollo$ git rev-parse HEAD 321bc25633fe2115e8ea4b2e68555c8c0d301b41 ~/playground/apollo$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE apolloauto/apollo dev-x86_64-20180320_1118 6a23927e28c3 7 days ago 6.92GB apolloauto/apollo yolo3d_volume-x86_64-latest 562d2b2b5a71 11 days ago 70.6MB apolloauto/apollo map_volume-sunnyvale_big_loop-latest 80aca30fa08a 3 weeks ago 1.3GB apolloauto/apollo localization_volume-x86_64-latest be947abaa650 2 months ago 5.74MB apolloauto/apollo map_volume-sunnyvale_loop-latest 36dc0d1c2551 2 months ago 906MB build cmd: in_dev_docker:/apollo$ ./apollo.sh build_no_perception dbg ``` 2. Compile pcl and copy the pcl library files to `/usr/local/lib`: See [/apollo/WORKSPACE.in](https://github.com/ApolloAuto/apollo/blob/master/WORKSPACE.in) to identify your pcl library version: - Prior to Apollo 5.0 (inclusive): pcl-1.7 - After Apollo 5.0: pcl-1.9 Inside docker: ``` (to keep pcl in host, we save pcl under /apollo) cd /apollo git clone https://github.com/PointCloudLibrary/pcl.git git checkout -b pcl- Ex: git checkout -b 1.7.2 pcl-1.7.2 git checkout -b 1.9.1 pcl-1.9.1 ``` then hack CMakeLists.txt with : ``` ~/playground/apollo/pcl$ git diff diff --git a/CMakeLists.txt b/CMakeLists.txt index f0a5600..42c182e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -7,6 +7,15 @@ endif() set(CMAKE_CONFIGURATION_TYPES "Debug;Release" CACHE STRING "possible configurations" FORCE) +if (CMAKE_VERSION VERSION_LESS "3.1") +# if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU") + set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11") + message("Build with c++11 support") +# endif () +else () + set (CMAKE_CXX_STANDARD 11) +endif () + # In case the user does not setup CMAKE_BUILD_TYPE, assume it's RelWithDebInfo if("${CMAKE_BUILD_TYPE}" STREQUAL "") set(CMAKE_BUILD_TYPE RelWithDebInfo CACHE STRING "build type default to RelWithDebInfo, set to Release to improve performance" FORCE) ``` Then build with: ``` cd pcl mkdir build cd build cmake .. make (We don't know the parameters that Apollo used, so we keep it by default) #backup pcl lib mkdir -p /usr/local/lib/pcl.origin mv /usr/local/lib/libpcl* /usr/local/lib/pcl.origin #replace with our build cp -a lib/* /usr/local/lib/ ldconfig ``` And finally restart Dreamview using ``` bash scripts/bootstrap.sh stop bash scripts/bootstrap.sh start ``` ### CPU does not support AVX instructions If CPU does not support AVX instructions, and you gdb the coredump file under /apollo/data/core/ with name core_dreamview.$PID, you may see logs like: ``` Program terminated with signal SIGILL, Illegal instruction. #0 0x000000000112b70a in std::_Hashtable, std::__detail::_Identity, std::equal_to, google::protobuf::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_Hashtable (this=0x3640288, __bucket_hint=10, __h1=..., __h2=..., __h=..., __eq=..., __exk=..., __a=...) ---Type to continue, or q to quit--- at /usr/include/c++/4.8/bits/hashtable.h:828 828 _M_rehash_policy() ``` To resolve this issue, in apollo/apollo.sh, comment or delete: ``` --copt=-mavx2 ``` Then try to build and start dreamview again.