core: api to save/get app settings to migrate them as part of the database (#3824)

* rfc: migrate app settings as part of export/import/migration

* export/import app settings

* test, fix

* chat: store app settings in db (#3834)

* chat: store app settings in db

* add combining with app-defaults

* commit schema

* test with tweaked settings

* remove unused error

---------

Co-authored-by: Evgeny Poberezkin <evgeny@poberezkin.com>

* remove app settings from export/import

* test, more settings

---------

Co-authored-by: Alexander Bondarenko <486682+dpwiz@users.noreply.github.com>
This commit is contained in:
Evgeny Poberezkin 2024-02-24 15:00:16 +00:00 committed by GitHub
parent b7709c59d3
commit e37654772f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
11 changed files with 334 additions and 3 deletions

View File

@ -0,0 +1,60 @@
# Migrating app settings to another device
## Problem
This is related to simplified database migration UX in the [previous RFC](./2024-02-12-database-migration.md).
Currently, when database is imported after the onboarding is complete, users can configure the app prior to the import.
Some of the settings are particularly important for privacy and security:
- SOCKS proxy settings
- Automatic image etc. downloads
- Link previews
With the new UX, the chat will start automatically, without giving users a chance to configure the app. That means that we have to migrate settings to a new device as well, as part of the archive.
## Solution
There are several possible approaches:
- put settings to the database via the API
- save settings as some file with cross-platform format (e.g. JSON or YAML or properties used on desktop).
The second approach seems much simpler than maintaining the settings in the database.
If we save a file, then there are two options:
- native apps maintain cross-platform schemas for this file, support any JSON and parse it in a safe way (so that even invalid or incorrect JSON - e.g., array instead of object - or invalid types in some properties do not cause the failure of properties that are correct).
- this schema and type will be maintained in the core library, that will be responsible for storing and reading the settings and passing to native UI as correct record of a given type.
The downside of the second approach is that addition of any property that needs to be migrated will have to be done on any change in either of the platforms. The downside of the first approach is that neither app platform will be self-sufficient any more, and not only iOS/Android would have to take into account code, but also each other code.
If we go with the second approach, there will be these types:
```haskell
data AppSettings = AppSettings
{ networkConfig :: NetworkConfig, -- existing type in Haskell and all UIs
privacyConfig :: PrivacyConfig -- new type, etc.
-- ... additional properties after the initial release should be added as Maybe, as all extensions
}
data ArchiveConfig = ArchiveConfig
{ -- existing properties
archivePath :: FilePath,
disableCompression :: Maybe Bool,
parentTempDirectory :: Maybe FilePath,
-- new property
appSettings :: AppSettings
-- for export, these settings will contain the settings passed from the UI and will be saved to JSON file as simplex_v1_settings.json in the archive
-- for import, these settings will contain the defaults that will be used if some property or subproperty is missing in JSON
}
-- importArchive :: ChatMonad m => ArchiveConfig -> m [ArchiveError] -- current type
importArchive :: ChatMonad m => ArchiveConfig -> m ArchiveImportResult -- new type
-- | CRArchiveImported {archiveErrors :: [ArchiveError]} -- current type
| CRArchiveImported {importResult :: ArchiveImportResult} -- new type
data ArchiveImportResult = ArchiveImportResult
{ archiveErrors :: [ArchiveError],
appSettings :: Maybe AppSettings
}
```

View File

@ -26,6 +26,7 @@ flag swift
library
exposed-modules:
Simplex.Chat
Simplex.Chat.AppSettings
Simplex.Chat.Archive
Simplex.Chat.Bot
Simplex.Chat.Bot.KnownContacts
@ -134,6 +135,7 @@ library
Simplex.Chat.Migrations.M20240115_block_member_for_all
Simplex.Chat.Migrations.M20240122_indexes
Simplex.Chat.Migrations.M20240214_redirect_file_id
Simplex.Chat.Migrations.M20240222_app_settings
Simplex.Chat.Mobile
Simplex.Chat.Mobile.File
Simplex.Chat.Mobile.Shared
@ -149,6 +151,7 @@ library
Simplex.Chat.Remote.Transport
Simplex.Chat.Remote.Types
Simplex.Chat.Store
Simplex.Chat.Store.AppSettings
Simplex.Chat.Store.Connections
Simplex.Chat.Store.Direct
Simplex.Chat.Store.Files

View File

@ -68,6 +68,7 @@ import Simplex.Chat.Protocol
import Simplex.Chat.Remote
import Simplex.Chat.Remote.Types
import Simplex.Chat.Store
import Simplex.Chat.Store.AppSettings
import Simplex.Chat.Store.Connections
import Simplex.Chat.Store.Direct
import Simplex.Chat.Store.Files
@ -597,6 +598,8 @@ processChatCommand' vr = \case
fileErrs <- importArchive cfg
setStoreChanged
pure $ CRArchiveImported fileErrs
APISaveAppSettings as -> withStore' (`saveAppSettings` as) >> ok_
APIGetAppSettings platformDefaults -> CRAppSettings <$> withStore' (`getAppSettings` platformDefaults)
APIDeleteStorage -> withStoreChanged deleteStorage
APIStorageEncryption cfg -> withStoreChanged $ sqlCipherExport cfg
TestStorageEncryption key -> sqlCipherTestKey key >> ok_
@ -6469,6 +6472,8 @@ chatCommandP =
"/db key " *> (APIStorageEncryption <$> (dbEncryptionConfig <$> dbKeyP <* A.space <*> dbKeyP)),
"/db decrypt " *> (APIStorageEncryption . (`dbEncryptionConfig` "") <$> dbKeyP),
"/db test key " *> (TestStorageEncryption <$> dbKeyP),
"/_save app settings" *> (APISaveAppSettings <$> jsonP),
"/_get app settings" *> (APIGetAppSettings <$> optional (A.space *> jsonP)),
"/sql chat " *> (ExecChatStoreSQL <$> textP),
"/sql agent " *> (ExecAgentStoreSQL <$> textP),
"/sql slow" $> SlowSQLQueries,

View File

@ -0,0 +1,190 @@
{-# LANGUAGE NamedFieldPuns #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE StrictData #-}
{-# LANGUAGE TemplateHaskell #-}
module Simplex.Chat.AppSettings where
import Control.Applicative ((<|>))
import Data.Aeson (FromJSON (..), (.:?))
import qualified Data.Aeson as J
import qualified Data.Aeson.TH as JQ
import Data.Maybe (fromMaybe)
import Data.Text (Text)
import Simplex.Messaging.Client (NetworkConfig, defaultNetworkConfig)
import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, enumJSON)
import Simplex.Messaging.Util (catchAll_)
data AppPlatform = APIOS | APAndroid | APDesktop deriving (Show)
data NotificationMode = NMOff | NMPeriodic | NMInstant deriving (Show)
data NotificationPreviewMode = NPMHidden | NPMContact | NPMMessage deriving (Show)
data LockScreenCalls = LSCDisable | LSCShow | LSCAccept deriving (Show)
data AppSettings = AppSettings
{ appPlatform :: Maybe AppPlatform,
networkConfig :: Maybe NetworkConfig,
privacyEncryptLocalFiles :: Maybe Bool,
privacyAcceptImages :: Maybe Bool,
privacyLinkPreviews :: Maybe Bool,
privacyShowChatPreviews :: Maybe Bool,
privacySaveLastDraft :: Maybe Bool,
privacyProtectScreen :: Maybe Bool,
notificationMode :: Maybe NotificationMode,
notificationPreviewMode :: Maybe NotificationPreviewMode,
webrtcPolicyRelay :: Maybe Bool,
webrtcICEServers :: Maybe [Text],
confirmRemoteSessions :: Maybe Bool,
connectRemoteViaMulticast :: Maybe Bool,
connectRemoteViaMulticastAuto :: Maybe Bool,
developerTools :: Maybe Bool,
confirmDBUpgrades :: Maybe Bool,
androidCallOnLockScreen :: Maybe LockScreenCalls,
iosCallKitEnabled :: Maybe Bool,
iosCallKitCallsInRecents :: Maybe Bool
}
deriving (Show)
defaultAppSettings :: AppSettings
defaultAppSettings =
AppSettings
{ appPlatform = Nothing,
networkConfig = Just defaultNetworkConfig,
privacyEncryptLocalFiles = Just True,
privacyAcceptImages = Just True,
privacyLinkPreviews = Just True,
privacyShowChatPreviews = Just True,
privacySaveLastDraft = Just True,
privacyProtectScreen = Just False,
notificationMode = Just NMInstant,
notificationPreviewMode = Just NPMMessage,
webrtcPolicyRelay = Just True,
webrtcICEServers = Just [],
confirmRemoteSessions = Just False,
connectRemoteViaMulticast = Just True,
connectRemoteViaMulticastAuto = Just True,
developerTools = Just False,
confirmDBUpgrades = Just False,
androidCallOnLockScreen = Just LSCShow,
iosCallKitEnabled = Just True,
iosCallKitCallsInRecents = Just False
}
defaultParseAppSettings :: AppSettings
defaultParseAppSettings =
AppSettings
{ appPlatform = Nothing,
networkConfig = Nothing,
privacyEncryptLocalFiles = Nothing,
privacyAcceptImages = Nothing,
privacyLinkPreviews = Nothing,
privacyShowChatPreviews = Nothing,
privacySaveLastDraft = Nothing,
privacyProtectScreen = Nothing,
notificationMode = Nothing,
notificationPreviewMode = Nothing,
webrtcPolicyRelay = Nothing,
webrtcICEServers = Nothing,
confirmRemoteSessions = Nothing,
connectRemoteViaMulticast = Nothing,
connectRemoteViaMulticastAuto = Nothing,
developerTools = Nothing,
confirmDBUpgrades = Nothing,
androidCallOnLockScreen = Nothing,
iosCallKitEnabled = Nothing,
iosCallKitCallsInRecents = Nothing
}
combineAppSettings :: AppSettings -> AppSettings -> AppSettings
combineAppSettings platformDefaults storedSettings =
AppSettings
{ appPlatform = p appPlatform,
networkConfig = p networkConfig,
privacyEncryptLocalFiles = p privacyEncryptLocalFiles,
privacyAcceptImages = p privacyAcceptImages,
privacyLinkPreviews = p privacyLinkPreviews,
privacyShowChatPreviews = p privacyShowChatPreviews,
privacySaveLastDraft = p privacySaveLastDraft,
privacyProtectScreen = p privacyProtectScreen,
notificationMode = p notificationMode,
notificationPreviewMode = p notificationPreviewMode,
webrtcPolicyRelay = p webrtcPolicyRelay,
webrtcICEServers = p webrtcICEServers,
confirmRemoteSessions = p confirmRemoteSessions,
connectRemoteViaMulticast = p connectRemoteViaMulticast,
connectRemoteViaMulticastAuto = p connectRemoteViaMulticastAuto,
developerTools = p developerTools,
confirmDBUpgrades = p confirmDBUpgrades,
iosCallKitEnabled = p iosCallKitEnabled,
iosCallKitCallsInRecents = p iosCallKitCallsInRecents,
androidCallOnLockScreen = p androidCallOnLockScreen
}
where
p :: (AppSettings -> Maybe a) -> Maybe a
p sel = sel storedSettings <|> sel platformDefaults <|> sel defaultAppSettings
$(JQ.deriveJSON (enumJSON $ dropPrefix "AP") ''AppPlatform)
$(JQ.deriveJSON (enumJSON $ dropPrefix "NM") ''NotificationMode)
$(JQ.deriveJSON (enumJSON $ dropPrefix "NPM") ''NotificationPreviewMode)
$(JQ.deriveJSON (enumJSON $ dropPrefix "LSC") ''LockScreenCalls)
$(JQ.deriveToJSON defaultJSON ''AppSettings)
instance FromJSON AppSettings where
parseJSON (J.Object v) = do
appPlatform <- p "appPlatform"
networkConfig <- p "networkConfig"
privacyEncryptLocalFiles <- p "privacyEncryptLocalFiles"
privacyAcceptImages <- p "privacyAcceptImages"
privacyLinkPreviews <- p "privacyLinkPreviews"
privacyShowChatPreviews <- p "privacyShowChatPreviews"
privacySaveLastDraft <- p "privacySaveLastDraft"
privacyProtectScreen <- p "privacyProtectScreen"
notificationMode <- p "notificationMode"
notificationPreviewMode <- p "notificationPreviewMode"
webrtcPolicyRelay <- p "webrtcPolicyRelay"
webrtcICEServers <- p "webrtcICEServers"
confirmRemoteSessions <- p "confirmRemoteSessions"
connectRemoteViaMulticast <- p "connectRemoteViaMulticast"
connectRemoteViaMulticastAuto <- p "connectRemoteViaMulticastAuto"
developerTools <- p "developerTools"
confirmDBUpgrades <- p "confirmDBUpgrades"
iosCallKitEnabled <- p "iosCallKitEnabled"
iosCallKitCallsInRecents <- p "iosCallKitCallsInRecents"
androidCallOnLockScreen <- p "androidCallOnLockScreen"
pure
AppSettings
{ appPlatform,
networkConfig,
privacyEncryptLocalFiles,
privacyAcceptImages,
privacyLinkPreviews,
privacyShowChatPreviews,
privacySaveLastDraft,
privacyProtectScreen,
notificationMode,
notificationPreviewMode,
webrtcPolicyRelay,
webrtcICEServers,
confirmRemoteSessions,
connectRemoteViaMulticast,
connectRemoteViaMulticastAuto,
developerTools,
confirmDBUpgrades,
iosCallKitEnabled,
iosCallKitCallsInRecents,
androidCallOnLockScreen
}
where
p key = v .:? key <|> pure Nothing
parseJSON _ = pure defaultParseAppSettings
readAppSettings :: FilePath -> Maybe AppSettings -> IO AppSettings
readAppSettings f platformDefaults =
combineAppSettings (fromMaybe defaultAppSettings platformDefaults) . fromMaybe defaultParseAppSettings
<$> (J.decodeFileStrict f `catchAll_` pure Nothing)

View File

@ -49,6 +49,7 @@ import Data.Word (Word16)
import Language.Haskell.TH (Exp, Q, runIO)
import Numeric.Natural
import qualified Paths_simplex_chat as SC
import Simplex.Chat.AppSettings
import Simplex.Chat.Call
import Simplex.Chat.Markdown (MarkdownList)
import Simplex.Chat.Messages
@ -245,6 +246,8 @@ data ChatCommand
| APIExportArchive ArchiveConfig
| ExportArchive
| APIImportArchive ArchiveConfig
| APISaveAppSettings AppSettings
| APIGetAppSettings (Maybe AppSettings)
| APIDeleteStorage
| APIStorageEncryption DBEncryptionConfig
| TestStorageEncryption DBEncryptionKey
@ -711,6 +714,7 @@ data ChatResponse
| CRChatError {user_ :: Maybe User, chatError :: ChatError}
| CRChatErrors {user_ :: Maybe User, chatErrors :: [ChatError]}
| CRArchiveImported {archiveErrors :: [ArchiveError]}
| CRAppSettings {appSettings :: AppSettings}
| CRTimedAction {action :: String, durationMilliseconds :: Int64}
deriving (Show)

View File

@ -0,0 +1,20 @@
{-# LANGUAGE QuasiQuotes #-}
module Simplex.Chat.Migrations.M20240222_app_settings where
import Database.SQLite.Simple (Query)
import Database.SQLite.Simple.QQ (sql)
m20240222_app_settings :: Query
m20240222_app_settings =
[sql|
CREATE TABLE app_settings (
app_settings TEXT NOT NULL
);
|]
down_m20240222_app_settings :: Query
down_m20240222_app_settings =
[sql|
DROP TABLE app_settings;
|]

View File

@ -562,6 +562,7 @@ CREATE TABLE note_folders(
favorite INTEGER NOT NULL DEFAULT 0,
unread_chat INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE app_settings(app_settings TEXT NOT NULL);
CREATE INDEX contact_profiles_index ON contact_profiles(
display_name,
full_name

View File

@ -0,0 +1,22 @@
{-# LANGUAGE OverloadedStrings #-}
module Simplex.Chat.Store.AppSettings where
import Control.Monad (join)
import Control.Monad.IO.Class (liftIO)
import qualified Data.Aeson as J
import Data.Maybe (fromMaybe)
import Database.SQLite.Simple (Only (..))
import Simplex.Chat.AppSettings (AppSettings (..), combineAppSettings, defaultAppSettings, defaultParseAppSettings)
import Simplex.Messaging.Agent.Store.SQLite (maybeFirstRow)
import qualified Simplex.Messaging.Agent.Store.SQLite.DB as DB
saveAppSettings :: DB.Connection -> AppSettings -> IO ()
saveAppSettings db appSettings = do
DB.execute_ db "DELETE FROM app_settings"
DB.execute db "INSERT INTO app_settings (app_settings) VALUES (?)" (Only $ J.encode appSettings)
getAppSettings :: DB.Connection -> Maybe AppSettings -> IO AppSettings
getAppSettings db platformDefaults = do
stored_ <- join <$> liftIO (maybeFirstRow (J.decodeStrict . fromOnly) $ DB.query_ db "SELECT app_settings FROM app_settings")
pure $ combineAppSettings (fromMaybe defaultAppSettings platformDefaults) (fromMaybe defaultParseAppSettings stored_)

View File

@ -99,6 +99,7 @@ import Simplex.Chat.Migrations.M20240104_members_profile_update
import Simplex.Chat.Migrations.M20240115_block_member_for_all
import Simplex.Chat.Migrations.M20240122_indexes
import Simplex.Chat.Migrations.M20240214_redirect_file_id
import Simplex.Chat.Migrations.M20240222_app_settings
import Simplex.Messaging.Agent.Store.SQLite.Migrations (Migration (..))
schemaMigrations :: [(String, Query, Maybe Query)]
@ -197,7 +198,8 @@ schemaMigrations =
("20240104_members_profile_update", m20240104_members_profile_update, Just down_m20240104_members_profile_update),
("20240115_block_member_for_all", m20240115_block_member_for_all, Just down_m20240115_block_member_for_all),
("20240122_indexes", m20240122_indexes, Just down_m20240122_indexes),
("20240214_redirect_file_id", m20240214_redirect_file_id, Just down_m20240214_redirect_file_id)
("20240214_redirect_file_id", m20240214_redirect_file_id, Just down_m20240214_redirect_file_id),
("20240222_app_settings", m20240222_app_settings, Just down_m20240222_app_settings)
]
-- | The list of migrations in ascending order by date

View File

@ -385,6 +385,7 @@ responseToView hu@(currentRH, user_) ChatConfig {logLevel, showReactions, showRe
CRChatError u e -> ttyUser' u $ viewChatError logLevel testView e
CRChatErrors u errs -> ttyUser' u $ concatMap (viewChatError logLevel testView) errs
CRArchiveImported archiveErrs -> if null archiveErrs then ["ok"] else ["archive import errors: " <> plain (show archiveErrs)]
CRAppSettings as -> ["app settings: " <> plain (LB.unpack $ J.encode as)]
CRTimedAction _ _ -> []
where
ttyUser :: User -> [StyledString] -> [StyledString]

View File

@ -14,6 +14,9 @@ import Data.Aeson (ToJSON)
import qualified Data.Aeson as J
import qualified Data.ByteString.Char8 as B
import qualified Data.ByteString.Lazy.Char8 as LB
import qualified Data.Text as T
import Simplex.Chat.AppSettings (defaultAppSettings)
import qualified Simplex.Chat.AppSettings as AS
import Simplex.Chat.Call
import Simplex.Chat.Controller (ChatConfig (..))
import Simplex.Chat.Options (ChatOpts (..))
@ -21,6 +24,7 @@ import Simplex.Chat.Protocol (supportedChatVRange)
import Simplex.Chat.Store (agentStoreFile, chatStoreFile)
import Simplex.Chat.Types (authErrDisableCount, sameVerificationCode, verificationCode)
import qualified Simplex.Messaging.Crypto as C
import Simplex.Messaging.Util (safeDecodeUtf8)
import Simplex.Messaging.Version
import System.Directory (copyFile, doesDirectoryExist, doesFileExist)
import System.FilePath ((</>))
@ -84,8 +88,9 @@ chatDirectTests = do
it "disabling chat item expiration doesn't disable it for other users" testDisableCIExpirationOnlyForOneUser
it "both users have configured timed messages with contacts, messages expire, restart" testUsersTimedMessages
it "user profile privacy: hide profiles and notificaitons" testUserPrivacy
describe "chat item expiration" $ do
it "set chat item TTL" testSetChatItemTTL
describe "settings" $ do
it "set chat item expiration TTL" testSetChatItemTTL
it "save/get app settings" testAppSettings
describe "connection switch" $ do
it "switch contact to a different queue" testSwitchContact
it "stop switching contact to a different queue" testAbortSwitchContact
@ -2195,6 +2200,24 @@ testSetChatItemTTL =
alice #$> ("/ttl none", id, "ok")
alice #$> ("/ttl", id, "old messages are not being deleted")
testAppSettings :: HasCallStack => FilePath -> IO ()
testAppSettings tmp =
withNewTestChat tmp "alice" aliceProfile $ \alice -> do
let settings = T.unpack . safeDecodeUtf8 . LB.toStrict $ J.encode defaultAppSettings
settingsApp = T.unpack . safeDecodeUtf8 . LB.toStrict $ J.encode defaultAppSettings {AS.webrtcICEServers = Just ["non-default.value.com"]}
-- app-provided defaults
alice ##> ("/_get app settings " <> settingsApp)
alice <## ("app settings: " <> settingsApp)
-- parser defaults fallback
alice ##> "/_get app settings"
alice <## ("app settings: " <> settings)
-- store
alice ##> ("/_save app settings " <> settingsApp)
alice <## "ok"
-- read back
alice ##> "/_get app settings"
alice <## ("app settings: " <> settingsApp)
testSwitchContact :: HasCallStack => FilePath -> IO ()
testSwitchContact =
testChat2 aliceProfile bobProfile $