github backfill cannot skip not found repositories any more

[syphar]

While testing for #1249 I stumbled on errors while backfilling the github repos.

Error is:

Error: error decoding response body: missing field `path` at line 1 column xxx

in a place where we actually filter out the NOT_FOUND errors to gracefully handle them.

I do know for that all the failures I saw that the repo didn't exist, but I don't know the exact response (yet), so which error it really is.

full backtrace

Stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /Users/syphar/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.48/src/backtrace/libunwind.rs:86
      backtrace::backtrace::trace_unsynchronized
             at /Users/syphar/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.48/src/backtrace/mod.rs:66
   1: backtrace::backtrace::trace
             at /Users/syphar/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.48/src/backtrace/mod.rs:53
   2: backtrace::capture::Backtrace::create
             at /Users/syphar/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.48/src/capture.rs:164
   3: backtrace::capture::Backtrace::new_unresolved
             at /Users/syphar/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.48/src/capture.rs:158
   4: failure::backtrace::internal::InternalBacktrace::new
             at /Users/syphar/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.8/src/backtrace/internal.rs:46
   5: failure::backtrace::Backtrace::new
             at /Users/syphar/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.8/src/backtrace/mod.rs:121
   6: <failure::error::error_impl::ErrorImpl as core::convert::From<F>>::from
             at /Users/syphar/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.8/src/error/error_impl.rs:19
   7: <failure::error::Error as core::convert::From<F>>::from
             at /Users/syphar/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.8/src/error/mod.rs:36
   8: docs_rs::utils::github_updater::GithubUpdater::graphql
             at src/utils/github_updater.rs:245
   9: docs_rs::utils::github_updater::GithubUpdater::load_repository
             at src/utils/github_updater.rs:138
  10: docs_rs::utils::github_updater::GithubUpdater::backfill_repositories
             at src/utils/github_updater.rs:108
  11: cratesfyi::DatabaseSubcommand::handle_args
             at src/bin/cratesfyi.rs:434
  12: cratesfyi::CommandLine::handle_args
             at src/bin/cratesfyi.rs:132
  13: cratesfyi::main
             at src/bin/cratesfyi.rs:21
  14: core::ops::function::FnOnce::call_once
             at /Users/syphar/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:227
  15: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /Users/syphar/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125
  16: std::rt::lang_start::{{closure}}
             at /Users/syphar/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src/rt.rs:66
  17: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/core/src/ops/function.rs:259
      std::panicking::try::do_call
             at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panicking.rs:381
      std::panicking::try
             at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panicking.rs:345
      std::panic::catch_unwind
             at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/panic.rs:396
      std::rt::lang_start_internal
             at /rustc/e1884a8e3c3e813aada8254edfa120e85bf5ffca/library/std/src/rt.rs:51
  18: std::rt::lang_start
             at /Users/syphar/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/src/rt.rs:65
  19: _main

[Nemo157]

It might be that

docs.rs/src/utils/github_updater.rs

Line 328 in 9fe8f02

path: Vec<GraphErrorPath>,

should be #[serde(default)], if it doesn't emit a path for errors attached to the root node.

[syphar]

@Nemo157 thanks for the hint, was thinking something like that.

(while we are using the path to decide if we bail! or return None)

[syphar]

when setting #[serde(default)] it seems to work, which lets me assume these errors without path are errors we ignore anyways (or errors github returns when also returning a repo).

Next error is (on https://github.com/rustsim/nalgebra)

Error: error decoding response body: missing field `data` at line 1 column 72

Where the repo seems to be moved / renamed

[syphar]

(the last one was rate limiting, only handled wrong)