MM-30026: Use DB master when getting team members from a session (#16170)

* MM-30026: Use DB master when getting team members from a session

A race condition happens when the read-replica isn't updated yet
by the time a session expiry message reaches another node in the cluster.

Here is the sequence of events that can cause it:
- Server1 gets any request which has to wipe session cache.

- The SQL query is written to DB master, and a cluster message is propagated
to clear the session cache for that user.

- Now before the read-replica is updated with the master’s update,
the cluster message reaches Server2. The session cache is wiped out for that user.

- _Any random_ request for that user hits Server2. Does NOT have to be
the update team name request. The request does not find the value
in session cache, because it’s wiped off, and picks it up from the DB.
Surprise surprise, it gets the stale value. Sticks it into the cache.

By now, the read-replica is updated. But guess what, we aren’t going to
ask the DB anymore, because we have it in the cache. And the cache has the stale value.

We use a temporary approach for now by introducing a context in the DB calls so that
the useMaster information can be easily passed. And this has the added advantage of
reusing the same context for future DB calls in case it happens. And we can also
add more context keys as needed.

A proper approach needs some architectural changes. See the issue for more details.

```release-note
Fixed a bug where a session will hold on to a cached value
in an HA setup with read-replicas configured.
```

* incorporate review comments

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

store/timerlayer/timerlayer.go

@@ -6425,10 +6425,10 @@ func (s *TimerLayerTeamStore) GetTeamsByUserId(userId string) ([]*model.Team, er
 	return result, err
 }
 
-func (s *TimerLayerTeamStore) GetTeamsForUser(userId string) ([]*model.TeamMember, error) {
+func (s *TimerLayerTeamStore) GetTeamsForUser(ctx context.Context, userId string) ([]*model.TeamMember, error) {
 	start := timemodule.Now()
 
-	result, err := s.TeamStore.GetTeamsForUser(userId)
+	result, err := s.TeamStore.GetTeamsForUser(ctx, userId)
 
 	elapsed := float64(timemodule.Since(start)) / float64(timemodule.Second)
 	if s.Root.Metrics != nil {