how_to_debug_dreamview_start_problem.md

## How to Debug the Dreamview Start Problem

### Steps to Start Dreamview

If you encounter problems when starting Dreamview in the `docker/scripts/dev` sequence, first check if you are using the correct commands as shown below.

```bash
$ bash docker/scripts/dev_start.sh
$ bash docker/scripts/dev_into.sh
$ bash apollo.sh build
$ bash scripts/bootstrap.sh
```
### Dreamview Fails to Start

If Dreamview fails to start, use the script below to check the Dreamview startup log and restart Dreamview.

```bash
# Start Dreamview in foreground to see any error message it prints out during startup
$ bash scripts/dreamview.sh start_fe

# check dreamview startup log
$ cat data/log/dreamview.out
terminate called after throwing an instance of 'CivetException'
  what():  null context when constructing CivetServer. Possible problem binding to port.

$ sudo apt-get install psmisc

# to check if dreamview is running from other terminal
$ sudo lsof -i :8888

# kill other running/pending dreamview
$ sudo fuser -k 8888/tcp

# restart dreamview again
$ bash scripts/dreamview.sh
```

### Debug dreamview with gdb

If you get nothing in dreamview startup logs, you can try to debug dreamview with gdb, use the following commands:

```
$ gdb --args /apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/dreamview/conf/dreamview.conf
# or
$ source scripts/apollo_base.sh;
$ start_gdb dreamview
```

Once gdb is launched, press `r` and `enter` key to run,  if dreamview crashes, then get the backtrace with `bt`.

### CPU does not support FMA/FMA3 instructions

If you see an error `Illegal instruction` and something related with **libpcl_sample_consensus.so.1.7** in gdb backtrace, then you probably need to rebuild pcl lib from source by yourself and replace the one in the docker.

This usually happens when you're trying to run Apollo/dreamview on a machine that the CPU does not support FMA/FMA3 instructions, it will fail because the prebuilt pcl lib shipped with docker image is compiled with FMA/FMA3 support.

There are 2 steps to resolve this issue:
1. Identify if the issue is due to pcl lib through gdb:
    find the coredump file under /apollo/data/core/ with name core_dreamview.$PID.
If you see logs like:
```
@in_dev_docker:/apollo$ gdb bazel-bin/modules/dreamview/dreamview  data/core/core_dreamview.378
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bazel-bin/modules/dreamview/dreamview...done.

warning: exec file is newer than core file.
[New LWP 378]
[New LWP 379]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: the debug information found in "/home/caros/secure_upgrade/depend_lib/libyaml-cpp.so.0.5.1" does not match "/home/caros/secure_upgrade/depend_lib/libyaml-cpp.so.0.5" (CRC mismatch).

Core was generated by `/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/dreamv'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x00007f79ebd32bec in double boost::math::detail::erf_inv_imp<double, boost::math::policies::policy<boost::math::policies::promote_float<false>, boost::math::policies::promote_double<false>, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> >(double const&, double const&, boost::math::policies::policy<boost::math::policies::promote_float<false>, boost::math::policies::promote_double<false>, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> const&, mpl_::int_<64> const*) () from /usr/local/lib/libpcl_sample_consensus.so.1.7
(gdb) bt
#0  0x00007f79ebd32bec in double boost::math::detail::erf_inv_imp<double, boost::math::policies::policy<boost::math::policies::promote_float<false>, boost::math::policies::promote_double<false>, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> >(double const&, double const&, boost::math::policies::policy<boost::math::policies::promote_float<false>, boost::math::policies::promote_double<false>, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> const&, mpl_::int_<64> const*) () from /usr/local/lib/libpcl_sample_consensus.so.1.7
#1  0x00007f79ebcf7f1e in _GLOBAL__sub_I_sac.cpp () from /usr/local/lib/libpcl_sample_consensus.so.1.7
#2  0x00007f79f3e1a2da in call_init (l=<optimized out>, argc=argc@entry=2, argv=argv@entry=0x7ffc7bc11c78, env=env@entry=0x7ffc7bc11c90) at dl-init.c:78
#3  0x00007f79f3e1a3c3 in call_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>) at dl-init.c:36
#4  _dl_init (main_map=0x7f79f402e1c8, argc=2, argv=0x7ffc7bc11c78, env=0x7ffc7bc11c90) at dl-init.c:126
#5  0x00007f79f3e0b29a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#6  0x0000000000000002 in ?? ()
#7  0x00007ffc7bc134bd in ?? ()
#8  0x00007ffc7bc134eb in ?? ()
#9  0x0000000000000000 in ?? ()
(gdb) q

@in_dev_docker:/apollo$ addr2line -C -f -e /usr/local/lib/libpcl_sample_consensus.so.1.7.2 0x375bec
double boost::math::detail::erf_inv_imp<double, boost::math::policies::policy<boost::math::policies::promote_float<false>, boost::math::policies::promote_double<false>, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> >(double const&, double const&, boost::math::policies::policy<boost::math::policies::promote_float<false>, boost::math::policies::promote_double<false>, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy, boost::math::policies::default_policy> const&, mpl_::int_<64> const*)
??:?
```

More info:
```
~/playground/apollo$ git rev-parse HEAD
321bc25633fe2115e8ea4b2e68555c8c0d301b41

~/playground/apollo$ docker image ls
REPOSITORY          TAG                                    IMAGE ID            CREATED             SIZE
apolloauto/apollo   dev-x86_64-20180320_1118               6a23927e28c3        7 days ago          6.92GB
apolloauto/apollo   yolo3d_volume-x86_64-latest            562d2b2b5a71        11 days ago         70.6MB
apolloauto/apollo   map_volume-sunnyvale_big_loop-latest   80aca30fa08a        3 weeks ago         1.3GB
apolloauto/apollo   localization_volume-x86_64-latest      be947abaa650        2 months ago        5.74MB
apolloauto/apollo   map_volume-sunnyvale_loop-latest       36dc0d1c2551        2 months ago        906MB

build cmd:
in_dev_docker:/apollo$ ./apollo.sh build_no_perception dbg
```
2. Compile pcl and copy the pcl library files to `/usr/local/lib`:

See [/apollo/WORKSPACE.in](https://github.com/ApolloAuto/apollo/blob/master/WORKSPACE.in) to identify your pcl library version:
- Prior to Apollo 5.0 (inclusive): pcl-1.7
- After Apollo 5.0: pcl-1.9

Inside docker:
```
(to keep pcl in host, we save pcl under /apollo)
cd /apollo
git clone https://github.com/PointCloudLibrary/pcl.git

git checkout -b <your pcl-lib version> pcl-<your pcl-lib version>
Ex: git checkout -b 1.7.2 pcl-1.7.2
    git checkout -b 1.9.1 pcl-1.9.1
```
then hack CMakeLists.txt with :
```
~/playground/apollo/pcl$ git diff
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f0a5600..42c182e 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -7,6 +7,15 @@ endif()

 set(CMAKE_CONFIGURATION_TYPES "Debug;Release" CACHE STRING "possible configurations" FORCE)

+if (CMAKE_VERSION VERSION_LESS "3.1")
+# if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
+    set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11")
+    message("Build with c++11 support")
+# endif ()
+else ()
+  set (CMAKE_CXX_STANDARD 11)
+endif ()
+
 # In case the user does not setup CMAKE_BUILD_TYPE, assume it's RelWithDebInfo
 if("${CMAKE_BUILD_TYPE}" STREQUAL "")
   set(CMAKE_BUILD_TYPE RelWithDebInfo CACHE STRING "build type default to RelWithDebInfo, set to Release to improve performance" FORCE)
```

Then build with:
```
cd pcl
mkdir build
cd build
cmake  ..
make
(We don't know the parameters that Apollo used, so we keep it by default)

#backup pcl lib
mkdir -p /usr/local/lib/pcl.origin
mv /usr/local/lib/libpcl* /usr/local/lib/pcl.origin

#replace with our build
cp -a lib/* /usr/local/lib/

ldconfig
```

And finally restart Dreamview using
```
    bash scripts/bootstrap.sh stop
    bash scripts/bootstrap.sh start
```

### CPU does not support AVX instructions

If CPU does not support AVX instructions, and you gdb the coredump file under /apollo/data/core/ with name core_dreamview.$PID, you may see logs like:

```
Program terminated with signal SIGILL, Illegal instruction.
#0  0x000000000112b70a in std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, google::protobuf::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::_Hashtable (this=0x3640288, __bucket_hint=10,
    __h1=..., __h2=..., __h=..., __eq=..., __exk=..., __a=...)
---Type <return> to continue, or q <return> to quit---
    at /usr/include/c++/4.8/bits/hashtable.h:828
828          _M_rehash_policy()
```

To resolve this issue, in apollo/apollo.sh, comment or delete:
```
--copt=-mavx2
```
Then try to build and start dreamview again.