Error log

 $ rm -rf seq2species_new_weights_small_short/
 $ DeepMicrobes.py --input_tfrec=combined_train_small_short.tfrec --model_name=seq2species --model_dir=seq2species_new_weights_small_short --max_len=400 --encode_method=one_hot &> error_redir.log
 2020-05-31 18:39:03.937254: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
I0531 18:39:03.939836 46912496414336 tf_logging.py:115] Using config: {'_model_dir': 'seq2species_new_weights_small_short', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 100000, '_save_checkpoints_secs': None, '_session_config': None, '_keep_checkpoint_max': 1000, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x2aaac308eef0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
W0531 18:39:03.940948 46912496414336 tf_logging.py:120] 'cpuinfo' not imported. CPU info will not be logged.
W0531 18:39:03.941206 46912496414336 tf_logging.py:120] 'psutil' not imported. Memory info will not be logged.
I0531 18:39:03.941266 46912496414336 tf_logging.py:115] Benchmark run: {'model_name': 'model', 'dataset': {'name': 'dataset_name'}, 'machine_config': {'gpu_info': {'count': 0}}, 'run_date': '2020-05-31T16:39:03.940321Z', 'tensorflow_version': {'version': '1.9.0', 'git_hash': 'v1.9.0-0-g25c197e023'}, 'tensorflow_environment_variables': [], 'run_parameters': [{'name': 'batch_size', 'long_value': 32}, {'name': 'train_epochs', 'long_value': 1}]}
I0531 18:39:03.993578 46912496414336 tf_logging.py:115] Calling model_fn.
I0531 18:39:04.419297 46912496414336 tf_logging.py:115] Done calling model_fn.
I0531 18:39:04.420130 46912496414336 tf_logging.py:115] Create CheckpointSaverHook.
I0531 18:39:04.617074 46912496414336 tf_logging.py:115] Graph was finalized.
I0531 18:39:11.244651 46912496414336 tf_logging.py:115] Running local_init_op.
I0531 18:39:11.252510 46912496414336 tf_logging.py:115] Done running local_init_op.
I0531 18:39:11.590389 46912496414336 tf_logging.py:115] Saving checkpoints for 0 into seq2species_new_weights_small_short/model.ckpt.
RUNNING MODE:  train
MAIN
INPUT TRAIN_FN one_hot
INPUT TRAIN_ONEHOT
PARSED_EXAMPLE {'label': <tf.Tensor 'ParseSingleExample/ParseSingleExample:0' shape=(?,) dtype=int64>, 'read': <tf.Tensor 'ParseSingleExample/ParseSingleExample:1' shape=(?,) dtype=float32>}
INPUTS BEFORE RESHAPE Tensor("IteratorGetNext:0", shape=(?, 1600), dtype=float32, device=/device:CPU:0)
INPUTS Tensor("reshape_input:0", shape=(?, 1, 400, 4), dtype=float32)
filter_dim (5, 1)
SHAPE (?, 1, 400, 4)
FILTERS <tf.Variable 'convolution_1/weights:0' shape=(1, 5, 4, 1) dtype=float32_ref>
INPUTS Tensor("convolution_1/Maximum:0", shape=(?, 1, 400, 84), dtype=float32)
filter_dim (9, 1)
SHAPE (?, 1, 400, 84)
FILTERS <tf.Variable 'convolution_2/weights:0' shape=(1, 9, 84, 1) dtype=float32_ref>
INPUTS Tensor("convolution_2/Maximum:0", shape=(?, 1, 400, 58), dtype=float32)
filter_dim (13, 1)
SHAPE (?, 1, 400, 58)
FILTERS <tf.Variable 'convolution_3/weights:0' shape=(1, 13, 58, 1) dtype=float32_ref>
INPUTS Tensor("convolution_3/Maximum:0", shape=(?, 1, 400, 180), dtype=float32)
filter_dim (400, 2828)
SHAPE (?, 1, 400, 180)
FILTERS <tf.Variable 'fc_1/weights:0' shape=(1, 400, 180, 2828) dtype=float32_ref>
INPUTS Tensor("fc_1/Maximum:0", shape=(?, 1, 1, 2828), dtype=float32)
filter_dim (1, 2828)
SHAPE (?, 1, 1, 2828)
FILTERS <tf.Variable 'fc_2/weights:0' shape=(1, 1, 2828, 2828) dtype=float32_ref>
LOGITS LABELS Tensor("output_layer/add:0", shape=(?, 2505), dtype=float32) Tensor("IteratorGetNext:1", shape=(?, ?), dtype=int64, device=/device:CPU:0)
Traceback (most recent call last):
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Tried to explicitly squeeze dimension 1 but dimension was not 1: 0
         [[Node: sparse_softmax_cross_entropy_loss/remove_squeezable_dimensions/Squeeze = Squeeze[T=DT_INT64, squeeze_dims=[-1], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorGetNext:1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/WUR/grosm002/DeepMicrobes_clone/DeepMicrobes.py", line 368, in <module>
    absl_app.run(main)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/absl/app.py", line 278, in run
    _run_main(main, args)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/absl/app.py", line 239, in _run_main
    sys.exit(main(argv))
  File "/home/WUR/grosm002/DeepMicrobes_clone/DeepMicrobes.py", line 360, in main
    train(flags.FLAGS, model_fn, 'dataset_name')
  File "/home/WUR/grosm002/DeepMicrobes_clone/DeepMicrobes.py", line 215, in train
    classifier.train(input_fn=input_fn_train, hooks=train_hooks)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 366, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1119, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1135, in _train_model_default
    saving_listeners)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1336, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 577, in run
    run_metadata=run_metadata)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1053, in run
    run_metadata=run_metadata)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1144, in run
    raise six.reraise(*original_exc_info)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1129, in run
    return self._sess.run(*args, **kwargs)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1201, in run
    run_metadata=run_metadata)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 981, in run
    return self._sess.run(*args, **kwargs)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Tried to explicitly squeeze dimension 1 but dimension was not 1: 0
         [[Node: sparse_softmax_cross_entropy_loss/remove_squeezable_dimensions/Squeeze = Squeeze[T=DT_INT64, squeeze_dims=[-1], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorGetNext:1)]]

Caused by op 'sparse_softmax_cross_entropy_loss/remove_squeezable_dimensions/Squeeze', defined at:
  File "/home/WUR/grosm002/DeepMicrobes_clone/DeepMicrobes.py", line 368, in <module>
    absl_app.run(main)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/absl/app.py", line 278, in run
    _run_main(main, args)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/absl/app.py", line 239, in _run_main
    sys.exit(main(argv))
  File "/home/WUR/grosm002/DeepMicrobes_clone/DeepMicrobes.py", line 360, in main
    train(flags.FLAGS, model_fn, 'dataset_name')
  File "/home/WUR/grosm002/DeepMicrobes_clone/DeepMicrobes.py", line 215, in train
    classifier.train(input_fn=input_fn_train, hooks=train_hooks)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 366, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1119, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1132, in _train_model_default
    features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1107, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/WUR/grosm002/DeepMicrobes_clone/DeepMicrobes.py", line 118, in model_fn
    logits=logits, labels=labels)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/ops/losses/losses_impl.py", line 856, in sparse_softmax_cross_entropy
    labels, logits, weights, expected_rank_diff=1)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/ops/losses/losses_impl.py", line 785, in _remove_squeezable_dimensions
    labels, predictions, expected_rank_diff=expected_rank_diff)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/ops/confusion_matrix.py", line 72, in remove_squeezable_dimensions
    labels = array_ops.squeeze(labels, [-1])
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
    return func(*args, **kwargs)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 2556, in squeeze
    return gen_array_ops.squeeze(input, axis, name)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 7946, in squeeze
    "Squeeze", input=input, squeeze_dims=axis, name=name)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
    op_def=op_def)
  File "/home/WUR/grosm002/miniconda2/envs/DeepMicrobes/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Tried to explicitly squeeze dimension 1 but dimension was not 1: 0
         [[Node: sparse_softmax_cross_entropy_loss/remove_squeezable_dimensions/Squeeze = Squeeze[T=DT_INT64, squeeze_dims=[-1], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorGetNext:1)]]