Tensorflow Object Detection API ValueError: No variables to save

Question

I'm trying to train a custom object detection model using tensorflow object detection api. For the training purposes I used pickled image data set for training and as model I used ssd_mobilenet_v1_coco. When I started the training it gave me this error.

Traceback (most recent call last):
  File "train.py", line 184, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "train.py", line 180, in main
    graph_hook_fn=graph_rewriter_fn)
  File "/content/models/research/object_detection/trainer.py", line 381, in train
    init_saver = tf.train.Saver(available_var_map)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1338, in __init__
    self.build()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1347, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1372, in _build
    raise ValueError("No variables to save")
ValueError: No variables to save

Full error code can find here...

[https://gist.github.com/mpgovinda/1f59f7de7873f6ec4c4426b79dc6827a][1]

How do I resolve this?

Answer 1

Tested Solution

Add fine_tune_checkpoint_type: "detection" to train_config: { ... } section in your *.config file.

Sample

train_config: {
  ...
  fine_tune_checkpoint: "./pre_trained_model/model.ckpt"
  fine_tune_checkpoint_type:  "detection"
  ...
}

Tested on ssd_mobilenet_v1_quantized_300x300_coco14_sync model.

Answer 2

Was training this model http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz

Ran into the same error. Don't really like others' solution of setting from_detection_checkpoint: false.

Was fixed by adding the load_all_detection_checkpoint_vars: true field in train_config of the pipeline.config that comes with the model.

I also added from_detection_checkpoint: true as it was not there, not sure if it was relevant to the solution.

fine_tune_checkpoint: "path_to_model_ckpt/model.ckpt"
from_detection_checkpoint: true
load_all_detection_checkpoint_vars: true # this here fixed it
num_steps: 200000 # irrelevant
fine_tune_checkpoint_type: "detection"

Answer 3

The latest models have this problem, to solve this:

Go to '.config' file of your model and in the training section change from_detection_checkpoint: true to false

It will work.

Happy Coding :)

Answer 4

I had this same error while trying to train faster-rcnn with inception v2. It turned out I had the wrong fine tune checkpoint in my model config file. After I linked to the appropriate checkpoint for the model the error went away.

Answer 5

I had the same error. Resolved by using a different checkpoint.

Answer 6

You should either use checkpoints or to remove "init_fn=init_fn" from slim.learning.train arguments and "initializer_fn" variable from trainer. you can access all these from line around 393 in trainer