yolov3 training coco2017
Created by: ChenYingpeng
Paddle version. `python Python 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import paddle print(paddle.version) 1.4.1
`
Run below code.
`python train.py --model_save_dir=save_models/yolov3/ --pretrain=weights/darknet53/ --data_dir=/home/ss/data/coco2017/ ----------- Configuration Arguments ----------- batch_size: 8 class_num: 80 data_dir: /home/ss/data/coco2017/ dataset: coco2017 debug: False draw_thresh: 0.5 image_name: None image_path: image input_size: 608 label_smooth: True learning_rate: 0.001 max_iter: 500200 model_save_dir: save_models/yolov3/ nms_posk: 100 nms_thresh: 0.45 nms_topk: 400 no_mixup_iter: 40000 pretrain: weights/darknet53/ random_shape: True snapshot_iter: 2000 start_iter: 0 syncbn: True use_gpu: True use_multiprocess: True valid_thresh: 0.005 weights: weights/yolov3
Found 2 CUDA devices.
W0527 10:26:51.076629 11907 device_context.cc:261] Please NOTE: device: 0, CUDA Capability: 61, Driver API Version: 9.1, Runtime API Version: 9.0
W0527 10:26:51.077989 11907 device_context.cc:269] device: 0, cuDNN Version: 7.0.
W0527 10:26:51.077998 11907 device_context.cc:293] WARNING: device: 0. The installed Paddle is compiled with CUDNN 7.3, but CUDNN version in your machine is 7.0, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version.
W0527 10:26:51.501081 11907 graph.h:204] WARN: After a series of passes, the current graph can be quite different from OriginProgram. So, please avoid using the OriginProgram()
method!
2019-05-27 10:26:51,502-WARNING:
You can try our memory optimize feature to save your memory usage:
# create a build_strategy variable to set memory optimize option
build_strategy = compiler.BuildStrategy()
build_strategy.enable_inplace = True
build_strategy.memory_optimize = True
# pass the build_strategy to with_data_parallel API
compiled_prog = compiler.CompiledProgram(main).with_data_parallel(
loss_name=loss.name, build_strategy=build_strategy)
!!! Memory optimize is our experimental feature !!!
some variables may be removed/reused internal to save memory usage,
in order to fetch the right value of the fetch_list, please set the
persistable property to true for each variable in fetch_list
# Sample
conv1 = fluid.layers.conv2d(data, 4, 5, 1, act=None)
# if you need to fetch conv1, then:
conv1.persistable = True
loading annotations into memory... Done (t=11.60s) creating index... index created! Load in 80 categories. I0527 10:27:04.616825 11907 build_strategy.cc:282] set enable_sequential_execution:1 I0527 10:27:07.029074 11907 build_strategy.cc:285] SeqOnlyAllReduceOps:0, num_trainers:1 Loaded 118287 images from coco2017. Loaded 118287 images from coco2017. Loaded 118287 images from coco2017. Loaded 118287 images from coco2017. Traceback (most recent call last): File "train.py", line 154, in train() File "train.py", line 125, in train fetch_list=[v.name for v in fetch_list]) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 580, in run return_numpy=return_numpy) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 446, in run_parallel exe.run(fetch_var_names, fetch_var_name) paddle.fluid.core.EnforceNotMet: Invoke operator concat error. Python Callstacks: File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/framework.py", line 1654, in append_op attrs=kwargs.get("attrs", None)) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(args, kwargs) File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/layers/tensor.py", line 214, in concat attrs={'axis': axis}) File "/home/ss/projects/models/PaddleCV/yolov3/models/yolov3.py", line 105, in build_model axis=1) File "train.py", line 46, in train model.build_model() File "train.py", line 154, in train() C++ Callstacks: Enforce failed. Expected out_dims[j] == ins[i][j], but received out_dims[j]:38 != ins[i][j]:24. Input tensors should have the same elements except the specify axis. at [/paddle/paddle/fluid/operators/concat_op.cc:66] PaddlePaddle Call Stacks: 0 0x7fcc3af74b68p void paddle::platform::EnforceNotMet::Initstd::string(std::string, char const, int) + 360 1 0x7fcc3af74eb7p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const, int) + 87 2 0x7fcc3b3b5027p paddle::operators::ConcatOp::InferShape(paddle::framework::InferShapeContext) const + 1511 3 0x7fcc3cc066a8p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const + 584 4 0x7fcc3cc06e64p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 292 5 0x7fcc3cc0478cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 332 6 0x7fcc3ca1302ap paddle::framework::details::ComputationOpHandle::RunImpl() + 250 7 0x7fcc3ca060e0p paddle::framework::details::OpHandleBase::Run(bool) + 160 8 0x7fcc3c96d7adp 9 0x7fcc3bce0df3p std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&) + 35 10 0x7fcc3b042c97p std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) + 39 11 0x7fccae52fa99p 12 0x7fcc3c96d342p 13 0x7fcc3b043fd4p ThreadPool::ThreadPool(unsigned long)::{lambda()#1 (closed)}::operator()() const + 404 14 0x7fcc6e272c80p 15 0x7fccae5286bap 16 0x7fccae25e41dp clone + 109
*** Aborted at 1558924039 (unix time) try "date -d @1558924039" if you are using GNU date *** *** Aborted at 1558924039 (unix time) try "date -d @1558924039" if you are using GNU date *** *** Aborted at 1558924039 (unix time) try "date -d @1558924039" if you are using GNU date *** *** Aborted at 1558924039 (unix time) try "date -d @1558924039" if you are using GNU date *** *** Aborted at 1558924039 (unix time) try "date -d @1558924039" if you are using GNU date *** PC: @ 0x0 (unknown) PC: @ 0x0 (unknown) PC: @ 0x0 (unknown) PC: @ 0x0 (unknown) PC: @ 0x0 (unknown) *** SIGTERM (@0x3e800002e83) received by PID 12008 (TID 0x7fc92e53d700) from PID 11907; stack trace: *** *** SIGTERM (@0x3e800002e83) received by PID 12014 (TID 0x7fc92e53d700) from PID 11907; stack trace: *** *** SIGTERM (@0x3e800002e83) received by PID 11996 (TID 0x7fc92e53d700) from PID 11907; stack trace: *** *** SIGTERM (@0x3e800002e83) received by PID 11993 (TID 0x7fc92e53d700) from PID 11907; stack trace: *** *** SIGTERM (@0x3e800002e83) received by PID 11999 (TID 0x7fc92e53d700) from PID 11907; stack trace: *** @ 0x7fccae532390 (unknown) @ 0x7fccae532390 (unknown) @ 0x7fccae532390 (unknown) @ 0x7fccae532390 (unknown) @ 0x4957ca (unknown) @ 0x4bc17e PyEval_EvalFrameEx @ 0x4954f1 PyDict_GetItem @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x499498 PyDict_GetItemString *** Aborted at 1558924039 (unix time) try "date -d @1558924039" if you are using GNU date *** @ 0x4a6165 (unknown) @ 0x4c1e6f PyEval_EvalFrameEx @ 0x500631 _PyArg_ParseTupleAndKeywords_SizeT @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x7fccae532390 (unknown) @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x4bc9b6 PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x55df16 (unknown) @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d55f3 (unknown) @ 0x4a577e PyObject_Call @ 0x4bed3d PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d54b9 (unknown) @ 0x4eebee (unknown) @ 0x4ee7f6 (unknown) @ 0x4aa9ab (unknown) @ 0x4c15bf PyEval_EvalFrameEx @ 0x7fcc7181c28c ImagingResampleHorizontal @ 0x4c136f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x7fccacae01ab (unknown) @ 0x7fccae2c9ac6 (unknown) @ 0x4c16e7 PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x7fccacb79109 (unknown) @ 0x4bc9b6 PyEval_EvalFrameEx @ 0x4c37ed PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx PC: @ 0x0 (unknown) @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x4bc9b6 PyEval_EvalFrameEx @ 0x4d4c9d (unknown) *** SIGTERM (@0x3e800002e83) received by PID 11990 (TID 0x7fc92e53d700) from PID 11907; stack trace: *** @ 0x55df16 (unknown) @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d55f3 (unknown) @ 0x4a577e PyObject_Call @ 0x4bed3d PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x7fcc7181c547 ImagingResample @ 0x4d54b9 (unknown) @ 0x4eebee (unknown) @ 0x4ee7f6 (unknown) @ 0x4aa9ab (unknown) @ 0x4c15bf PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x7fccacae03fa (unknown) @ 0x7fccaca7ddd5 (unknown) @ 0x7fccae532390 (unknown) @ 0x7fcc7181686c (unknown) @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c16e7 PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c16e7 PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c16e7 PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x4bc9b6 PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x55df16 (unknown) @ 0x4bc3fa PyEval_EvalFrameEx @ 0x7fccacae0273 (unknown) @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x7fccacb69ce4 (unknown) @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d55f3 (unknown) @ 0x4a577e PyObject_Call @ 0x4bed3d PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d54b9 (unknown) @ 0x4eebee (unknown) @ 0x4ee7f6 (unknown) @ 0x4aa9ab (unknown) @ 0x4c15bf PyEval_EvalFrameEx @ 0x7fccae24e27d __read @ 0x7fccacae2166 (unknown) @ 0x7fccacb6f329 (unknown) @ 0x4bd2c4 PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x4bc9b6 PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x55df16 (unknown) @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x7fccae1d15e8 _IO_file_underflow @ 0x4d55f3 (unknown) @ 0x4a577e PyObject_Call @ 0x4bed3d PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d54b9 (unknown) @ 0x4eebee (unknown) @ 0x4ee7f6 (unknown) @ 0x4aa9ab (unknown) @ 0x4c15bf PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x7fccacb79178 (unknown) @ 0x4c37ed PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x4bc9b6 PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x55df16 (unknown) @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d55f3 (unknown) @ 0x4a577e PyObject_Call @ 0x4bed3d PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d54b9 (unknown) @ 0x7fccae1d0058 (unknown) @ 0x4eebee (unknown) @ 0x4ee7f6 (unknown) @ 0x4aa9ab (unknown) @ 0x4c15bf PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x7fccae1c5236 fread @ 0x7fcc99a00b62 (unknown) @ 0x7fcc99a011fd (unknown) *** Aborted at 1558924039 (unix time) try "date -d @1558924039" if you are using GNU date *** @ 0x7fcc99a01986 cv::imread() PC: @ 0x0 (unknown) @ 0x7fcc9f64ccb4 (unknown) @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x4bc9b6 PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x55df16 (unknown) @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx *** SIGTERM (@0x3e800002e83) received by PID 12005 (TID 0x7fc92e53d700) from PID 11907; stack trace: *** @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d55f3 (unknown) @ 0x4a577e PyObject_Call @ 0x4bed3d PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d54b9 (unknown) @ 0x4eebee (unknown) @ 0x4ee7f6 (unknown) @ 0x4aa9ab (unknown) @ 0x4c15bf PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x7fccae532390 (unknown) @ 0x7fccae24e27d __read @ 0x7fccae1cff79 (unknown) *** Aborted at 1558924039 (unix time) try "date -d @1558924039" if you are using GNU date *** @ 0x7fccae1c5236 fread PC: @ 0x0 (unknown) *** SIGTERM (@0x3e800002e83) received by PID 12011 (TID 0x7fc92e53d700) from PID 11907; stack trace: *** @ 0x7fcc715c18f4 (unknown) @ 0x7fccae532390 (unknown) @ 0x7fcc715c5c0a jpeg_fill_bit_buffer @ 0x7fcc715c6797 (unknown) @ 0x7fccae24e27d __read @ 0x7fcc715c2186 (unknown) @ 0x7fccae1d15e8 _IO_file_underflow @ 0x7fcc715c7cde (unknown) @ 0x7fccae1d0058 (unknown) @ 0x7fcc715c129a jpeg_read_scanlines @ 0x7fccae1c5236 fread @ 0x7fcc99a1b3ce cv::JpegDecoder::readData() @ 0x7fcc99a00b62 (unknown) @ 0x7fcc99a011fd (unknown) @ 0x7fcc99a01449 (unknown) @ 0x7fcc99a01986 cv::imread() @ 0x7fcc99a01986 cv::imread() @ 0x7fcc9f64ccb4 (unknown) @ 0x7fcc9f64ccb4 (unknown) @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x4d4c9d (unknown) @ 0x4bc9b6 PyEval_EvalFrameEx @ 0x4bc9b6 PyEval_EvalFrameEx @ 0x4d4c9d (unknown) @ 0x4d4c9d (unknown) @ 0x55df16 (unknown) @ 0x55df16 (unknown) @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4bc3fa PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4c1e6f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d55f3 (unknown) @ 0x4d55f3 (unknown) @ 0x4a577e PyObject_Call @ 0x4a577e PyObject_Call @ 0x4bed3d PyEval_EvalFrameEx @ 0x4bed3d PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4b9ab6 PyEval_EvalCodeEx @ 0x4d54b9 (unknown) @ 0x4d54b9 (unknown) @ 0x4eebee (unknown) @ 0x4ee7f6 (unknown) @ 0x4aa9ab (unknown) @ 0x4c15bf PyEval_EvalFrameEx @ 0x4c136f PyEval_EvalFrameEx `
How to solve this problem? Thank you in advance.