[TNTP-8909] fix: stop following the transfer destination on bucket ref error retries

[Satbek]

During rebalancing the source storage rejects a request with a bucket
ref error (e.g. TRANSFER_IS_IN_PROGRESS) and reports the transfer
destination. The router followed this redirect and retried the request
directly on the destination.

The destination, however, may not have received the bucket yet: it has
neither the _bucket record nor the data, and it is still in fast mode,
so no bucket ownership check is performed. The request was applied
against an empty space and acknowledged to the client. Once
bucket_recv() delivered the data, the cluster state diverged from the
acknowledged result: a deleted tuple reappeared, and the final result
of replace/update/upsert could differ from what was acknowledged.

The router no longer follows the destination reported by the error:
the bucket route cache is reset and the retry target is re-discovered
via vshard discovery, so a request is applied only on the replicaset
that actually holds the bucket.

A failed request is retried at most once and only after a recovery
action: the master is located on MISSING_MASTER, the cached master is
updated on NON_MASTER, the bucket route is reset on bucket ref errors
and the retry is re-routed by the bucket id. Other errors are returned
to the caller without a retry: the old code retried them blindly,
which silently doubled the effective request timeout. The retry is
given only the time left before the request deadline. Map requests are
not recovered at all: they are async, so storage errors arrive later,
in the future payload (see #513).

The rebalance safe mode also missed the start of a transfer on vshard
0.1.41+, where the bucket status is UPDATEd to READONLY instead of
being REPLACEd with SENDING, so safe mode was not enabled during
rebalancing and writes were not protected with bucket refs. Now the
READONLY update enables the safe mode as well.

Closes #509

[Serpentian]

Great work!

[Serpentian]

Almost there! We're moving fast

Please, let's properly split the changes between the commits. Otherwise, I cannot judge, whether the code is properly splitted and I am forced to look at all the changes in their entirety, this way it's way easier to skip the bug.

And if you don't mind, can you in the future please write your responses to each comment before re-requesting their review? This helps a lot to see if you have addressed the comments. Even a simple "Fixed" is better than nothing. Thanks in advance!

[Serpentian]

Looks like the last iteration to me. Thank you for putting up with my whining :)

[Satbek]

I also returned back recovery for call.map because we actually can handle MISSING_MASTER error in it.

[Serpentian]

Thank you for the fixes, the current version seems correct. Speaking of existing code, it requires substantial refactoring, it's very hard to make changes to it now: there're too many rough edges and small bugs in it. But I really appreciate, that you've fixed the things we've found.

I really hope, that someday we'll drop all that code from crud and allow vshard to do its work in order to avoid such nasty bugs. But for that to happen we must find a way to bump the perf of all requests