IntenseFinance/simplex-chat
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update github content (#519)

Browse Source 
* update github content

* update comparison

* update link

* move message_views.sql to scripts

* move section

* move news section

* typos

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>

* update readme

* update readme

* update readme

Co-authored-by: JRoberts <8711996+jr-simplex@users.noreply.github.com>
This commit is contained in:
Evgeny Poberezkin
2022-04-11 22:29:08 +01:00
committed by GitHub
parent 0a17f5c491
commit af471d0077
12 changed files with 377 additions and 341 deletions
Download Patch File Download Diff File Expand all files Collapse all files

167
docs/rfcs/2022-01-26-mobile-app.md Normal file
View File

@@ -0,0 +1,167 @@
# Porting SimpleX Chat to mobile
## Background and motivation
We have code that "works", the aim is to keep platform differences in the core minimal and get the apps to market faster.
### SimpleX platform design
See [overview](https://github.com/simplex-chat/simplexmq/blob/master/protocol/overview-tjr.md) for overall platform design and objectives, it is worth reading the introduction. The diagram copied from this doc:
```
User's Computer Internet Third-Party Server
------------------ | ---------------------- | -------------------------
| |
SimpleX Chat | |
| |
+----------------+ | |
| Chat App | | |
+----------------+ | |
| SimpleX Agent | | |
+----------------+ -------------- TLS ---------------- +----------------+
| SimpleX Client | ------ SimpleX Messaging Protocol ------> | SimpleX Server |
+----------------+ ----------------------------------- +----------------+
| |
```
- SimpleX Servers only pass messages, we don't need to touch that for the app
- SimpleX clients talk to the servers, we won't use them directly
- SimpleX agent is used from chat, we won't use it directly from the app
- Chat app will expose API to the app to communicate with everything, including DB and network.
### Important application modules
Modules of simplexmq package used from simplex-chat:
- a [functional API in Agent.hs]([Agent.hs](https://github.com/simplex-chat/simplexmq/blob/master/src/Simplex/Messaging/Agent.hs#L38)) to send messages and commands
- TBQueue to receive messages and notifications (specifically, [subQ field of AgentClient record in Agent/Client.hs](https://github.com/simplex-chat/simplexmq/blob/master/src/Simplex/Messaging/Agent/Client.hs#L72))
- [types from Agent/Protocol.hs](https://github.com/simplex-chat/simplexmq/blob/master/src/Simplex/Messaging/Agent/Protocol.hs)).
This package has its [own sqlite database file](https://github.com/simplex-chat/simplexmq/tree/master/migrations) - as v1 was not backwards compatible migrations are restarted - where it stores all encryption and signing keys, shared secrets, servers and queue addresses - effectively it completely abstracts the network away from chat application, providing an API to manage logical duplex connections.
Simplex-chat library is what we will use from the app:
- command type [ChatCommand in Chat.hs](https://github.com/simplex-chat/simplex-chat/blob/master/src/Simplex/Chat.hs#L72) that UI can send to it
- UI sends these commands via TBQueue that `inputSubscriber` reads in forever loop and sends to `processChatCommand`. There is a hack that `inputSubscriber` not only reads commands but also shows them in the view, depending on the commands.
- collection of [view functions in Chat/View.hs](https://github.com/simplex-chat/simplex-chat/blob/master/src/Simplex/Chat/View.hs) to reflect all events in view.
This package also creates its own [database file](https://github.com/simplex-chat/simplex-chat/tree/master/migrations) where it stores references to agent connections managed by the agent, and how they map to contacts, groups, and file transmissions.
## App design options and questions
### Sending chat commands from UI and receiving them in Haskell
Possible options:
- function (exported via FFI) that receives strings from UI and decodes them into ChatCommand type, then sending this command to `processChatCommand`. This option requires a single function in C header file, but also requires encoding in UI and decoding in Haskell.
- multiple functions exported via FFI each sending different command to `processChatCommand`. This option requires multiple functions in header file and multiple exports from Haskell.
Overall, the second option seems a bit simpler and cleaner, if we agree to go this route we will refactor `processChatCommand` to expose its parts that process different commands as independent functions.
On another hand, it might be easier to grow chat API if this is passed via a single function and serialized as strings (e.g. as JSON, to have it more universal) - it would also might give us an API for a possible future chat server that works with thin, UI-only clients.
In both cases, we should split `processChatCommand` (or the functions it calls) into a separate module, so it does not have code that is not used from the app.
**Proposal**
Use option 2 to send commands from UI to chat, encoding/decoding commands as strings with a tag in the beginning (TBC binary, text or JSON based - encoding will have to be replicated in UI land; both encoding and decoding is needed in Haskell land to refactor terminal chat to use this layer as well, so we have a standard API for all implementations).
This function would have this type:
```haskell
sendRequest :: CString -> IO CString
```
to allow instant responses.
One more idea. This function could be made to match REST semantics that would simplify making chat into a REST chat server api:
```haskell
sendRequest :: CString -> CString -> CString -> CString -> IO CString
sendRequest verb path qs body = pure ""
```
### Sending messages and notifications from Haskell to UI
Firstly, we have to refactor the existing code so that all functions in [View.hs](https://github.com/simplex-chat/simplex-chat/blob/master/src/Simplex/Chat/View.hs) are passed to `processChatCommand` (or the functions for each command, if we go with this approach) as a single record containing all view functions.
The current code from View.hs will not be used in the mobile app, it is terminal specific; we will create a separate connector to the UI that has the same functions in a record - these functions communicate to the UI.
Again, there are two similar options how this communication can happen:
- UIs would export multiple functions however each platform allows it, as C exports, and they would be all imported in Haskell. This option feels definitely worse, as it would have to be maintained in both iOS and Android separately for exports, and in Haskell for imports, resulting in lots of boilerplate.
- UIs would export one function that receives strings (e.g. JSON encoded) with the messages and notifications, there will be one function in Haskell to send these JSON. All required view functions in Haskell land would simply send different strings into the same function.
In this case the second option seems definitely easier, as even with simple terminal UI there are more view events than chat commands (although, given different mobile UI paradigms some of these events may not be needed, but some additional events are likely to be addedd, that would be doing nothing for terminal app).
**Proposal**
Encode messages and notifications as JSON, but instead of exporting the function from UI (which would have to be done differently from different platforms), have Haskell export function `receiveMessage` that would be blocking until the next notification or message is available. UI would handle it in a simple loop, on a separate thread:
```haskell
-- CString is serialized JSON (ToJSON serialized datatype from haskell)
receiveMessage :: IO CString ()
```
To convert between Haskell and C interface:
```haskell
type CJSON = CString
toCJSON ToJSON a => a -> CJSON
toCJSON = ...
-- Haskell interface
send :: ToJSON a => String -> IO a
recv :: ToJSON a => IO a
-- C interface
c_send :: CString -> IO CJSON
c_recv :: IO CJSON
```
### Accessing chat database from the UI
Unlike terminal UI that does not provide any capabilities to access chat history, mobile UI needs to have access to it.
Two options how it can be done:
- UI accesses database directly via its own database library. The upside of this approach is that it keeps Haskel core smaller. The downside is that sqlite is relatively bad with concurrent access. In Haskell code we allowed some concurrency initially, having the pool limited to few concurrent connection, but later we removed concurrency (by limiting pool size to 1), as otherwise it required retrying to get transaction locks with difficult to set retry time limits, and leading to deadlocks in some cases. Also mobile sqlite seems to be compiled with concurrency disabled, so we would have to ship app with our own sqlite (which we might have to do anyway, for the sake of full text search support). We could use some shared semaphore in Haskell to obtain database lock, but it adds extra complexity...
- UI accesses database via Haskell functions. The upside of this is that there would be no issues with concurrency, and chat schema would be "owned" by Haskell core, but it requires either a separate serializable protocol for database access or multiple exported functions (same two options as before).
However bad the second option is, it seems slightly better as at least we would not have to duplicate sql quiries in iOS and Android. But this is the trade-off I am least certain of...
**Proposal**
Use the same `sendRequest` function to access database.
Additional idea: as these calls should never mutate chat database, they should only query the state, and as these functions will not be needed for terminal UI, I think we could export it as a separate function and have all necessary queries/functions in a separate module, e.g.:
```haskell
-- params and result are JSON encoded
chatQuery :: CString -> IO CString
chatQuery params = pure ""
```
On another hand, if we go with REST-like `sendRequest` then it definitely should be the only function to access chat and database state.
### UI database
UI needs to have its own storage to store information about user settings in the app and, possibly, which chat profiles the user has (each would have its own chat/agent databases).
### Chat database initialization
Currently it is done in an ad hoc way, during the application start ([`getCreateActiveUser` function](https://github.com/simplex-chat/simplex-chat/blob/master/src/Simplex/Chat.hs#L1178)), we could either expose this function to accept database name or just check on the start and initialize database with the default name in case it is not present.
### Multiple profiles in the app
All user profiles are stored in the same database. The current schema allows multiple profiles, but the current UI does not. We do not need to do it in the app MVP.
## Notifications
We don't need it in the first version - it is out of scope of releasable MVP - but we need to think a bit ahead how it will be done so it doesn't invalidate the design we settle on.
There is no reliable background execution, so the only way to receive messages when the app is off is via notifications. We have added notification subscriptions to the low protocol layer so that Haskell core would receive function call when notification arrives to the native part and receive and process messages and communicate back to the local part that would show a local notification on the device:
```
Push notification -> Native -> Haskell ... process ... -> Native -> Local notification
```
Notifications are the main reason why we will need to store multiple profiles in the same database file - when notification arrives we do not know which profile it is for, it only has server address and queue ID, and if different profiles were in different databases we would either had to have a single table mapping queues to profiles or lookup multiple databases - both options seem worse than a single database with multiple profiles.
For the rest we would just use the same approaches we would use for UI/Haskell communications - probably a separate functions to receive notifications to Haskell, and the same events to be sent back.

19
docs/rfcs/2022-02-10-deduplicate-contact-requests.md Normal file
View File

@@ -0,0 +1,19 @@
# Deduplicate contact requests
1. add nullable fields `via_contact_uri_hash` and `xcontact_id` to `connections`
2. when joining (Connect -> SCMContact)
- generate and save random `xcontact_id`
- save hash of `AConnectionRequestUri` when joining via contact uri
(AConnectionRequestUri -> ConnectionRequestUri -> CRContactUri)
- send random identifier in `XContact` as `Maybe XContactId`
- check for repeat join - if connection with such `via_contact_uri_hash` has contact notify user
- check for repeat join - check in connections if such contact uri exists, if yes use same identifier; the rest of request can (should) be regenerated, e.g. new server, profile
can be required
3. add nullable field `xcontact_id` to `contact_requests` and to `contacts` (* for auto-acceptance)
4. on contact request (processUserContactRequest)
- save identifier
- \* check if `xcontact_id` is in `contacts` - then notify this contact already exists
- when saving check if contact request with such identifier exists, if yes update `contact_request`
(`invId`, new profile)
- ? remove old invitation - probably not necessarily, to be done in scope of connection expiration
- return from Store whether request is new or updated (Bool?), new chat response for update or same response

29
docs/rfcs/2022-02-24-servers-configuration.md Normal file
View File

@@ -0,0 +1,29 @@
# Server configuration
- in agent:
- Agent.Env.SQLite - move smpServers from AgentConfig to Env, make it TVar; keep "initialSmpServers" in AgentConfig?
- Agent - getSMPServer to read servers from Env and choose a random server
- Agent - new functional api - "useServers"
- ~~Agent.Protocol - new ACommand?~~
- chat core:
- db:
- new table `smp_servers`, server per row, same columns as for agent. Have rowid for future
- getServers method
- update - truncate and rewrite
- ChatCommand GetServers - new ChatResponse with list of user SMPServers, it may be empty if default are used
- ChatCommand SetServers - ChatResponse Ok (restore default servers is empty set servers list)
- agent config is populated using getServers, if it's empty default are used
- mobile chat:
- mobileChatOpts to be populated with initial servers on init (getServers or default if empty)
- in ui:
- view in settings
- GetServers on view open to populate
- Confirm buttons, Restore button - destructive - clears user servers and default are used
- validation
- validation on submit, error with server's string
- ~~TBD real-time validation~~
- ~~fastest is validation on submit without detailed error?~~
- ~~maybe even faster - alternatively have 3 fields for entry per server - fingerprint, host, port - and build server strings (still validate to avoid hard crash?)?~~
- terminal chat:
- if -s option is given, these servers are used and getServers is not used for populating agentConfig
- if -s option is not provided - same as in mobile - getServers or default if empty

14
docs/rfcs/2022-03-02-avatars.md Normal file
View File

@@ -0,0 +1,14 @@
# Include (Optional) Images in User Profiles
1. Add SQL migration for database in `src/Simplex/Chat/Migrations`
- This will touch `contact_profiles` and `group_profiles`
2. Add field to `User` in `Types.hs` allowing for null entry using `Maybe`
3. Extend parsing in `Chat.hs` under `chatCommandP :: Parser ChatCommand`
4. Update `UpdateProfile` in `Chat.hs` to accept possible display picture and implement an `APIUpdateProfile` command which accepts a JSON string `/_profile{...}` which will add the image to a profile.
5. Connect up to Android and iOS apps (new PRs)
Profile images will be base 64 encoded images. We can use the `base64P` parser to process them and pass them as JSON.

61
docs/rfcs/2022-03-02-number-chat-items.md Normal file
View File

@@ -0,0 +1,61 @@
# Message replies and chat item sequential numbers
## Problem
Many chat features require referring to the previous chat items in the same conversation:
- item editing
- item deletion
- item reply (with quoting)
- delivery/read receipts
- any interactive features mutating chat item state
- group message integrity via DAG
The most in-demand feature is replies.
## Proposed solution
As group message integrity is needed not for chat items, but for messages, the updated proposal is to introduce a random, non-sequential message id, unique per conversation and per sender.
All above features would rely on this ID, e.g. reply would use the ID of the message that created the item.
We will add an optional property `msgId` into all chat messages (not only visible to the users) and `msgRef` into messages that need to reference other messages.
`msgId` property is a base64 encoded 12 byte binary
JTD for quoting messages:
```yaml
definitions:
msgRef:
discriminator: type
mapping:
direct:
properties:
msgId: type: string
sentAt: type: datetime
sent: type: boolean # true if it is in reference to the item that the sender of the message originally sent, false for references to received items
group:
properties:
msgId: type: string
sentAt: type: datetime
memberId: type: string # base64 member ID of the sender known to all group members for group chats
content:
properties:
type: type: string
text: type: string
properties:
msgId: string
event: enum: ["x.msg.new"]
params:
properties:
content: ref: content
quote:
properties:
content: ref: content
msgRef: ref: msgRef
```
This format ensures that replies with quoting show as normal messages on the clients that do not support showing quotes (`quote` property will be ignored).
The only feature that would not work in case chatItem/chatItemRef is missing is navigating to the message to which the message is in reply to.