Commit Graph

63 Commits

Author SHA1 Message Date
Jesús Espino
27d536b212 MM-21552: Adding SaveMultiple to posts (#13766)
* Adding SaveMultiple to posts

* Improving tests

* fixing i18n

* Fixing tests

* Improving testing on top of Save and SaveMultiple

* Fixing shadow variables

* Addressing some PR comments

* More clear update post test

* Addressing some PR comments

* Addressing some PR comments and simplifying the code

* Improting replies in bulk too

* Fixing reply count and processing last imported replies

* Adding OverwriteMultiple to posts aggregating everything in the same transaction

* Adding 2 pending tests to implement

* Adding tests for overwrite multiple posts

* Adding tests for TeamStore.GetByNames method

* Fixing shadow variables

* Addressing PR comments

* Extracting i18n strings

* Fixing tests

* Fixing tests

* Adding more test cases

* Using a variable instead of a fake timestamp
2020-03-11 14:29:32 +01:00
Ibrahim Serdar Acikgoz
d5df6a3fc4 [MM-22314] Remove read limit at app/import which cause bulk import to fail (#13863)
* app/import: increase buffer size to avoid failing in large inputs

* remove commented out lock

* app/import: make maxTokenSize constant

Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-03-06 23:14:35 +03:00
Eli Yukelzon
17523fa5d9 MM-21898 - Part 1: Generate and use an interface instead of *A… (#13840)
* Generate and use an interface instead of *App
2020-02-13 13:26:58 +01:00
Jesús Espino
ae6f15b76f Making private import methos in the app layer (#13758)
* Making private import methos in the app layer

* Making one more method private
2020-01-28 10:58:33 +01:00
Jesús Espino
a63684fcb5 Consistent license message for all the go files (#13235)
* Consistent license message for all the go files

* Fixing the last set of unconsistencies with the license headers

* Addressing PR review comments

* Fixing busy.go and busy_test.go license header
2019-11-29 12:59:40 +01:00
Miguel de la Cruz
2259b7f2a8 [MM-19948] Set version on module file and internal paths (#13186)
* [MM-19948] Set version on module file and internal paths

* Fixes after merge

* Fix i18n checker error
2019-11-28 14:39:38 +01:00
Ben Sooraj
c7e2689926 mlog standardisation for app/import.go (#12741)
* mlog standardisation for app/import.go

* using the corrent mlog.Field type for the error
2019-10-17 11:18:08 +01:00
Jesús Espino
c0b51b03de MM-16548: Fixing bug with empty type in the second line of the bulk import (#11567)
* MM-16548: Fixing bug with empty type in the second line of the bulk import

* Adding tests to this behavior

* Changing the approach to fix the problem
2019-07-08 16:17:06 +01:00
Shota Gvinepadze
fab2e349b3 [MM-10768] Migrate "Channel.SaveDirectChannel" to Sync by default (#10846) 2019-05-16 22:04:58 +02:00
Siyuan Liu
27566f6a06 [MM-13033] Continue bulk import on large image error (#10780)
* [MM-13033] Continue bulk import on large image error

* add tests, review comments

* Update app/import_functions.go

Co-Authored-By: liusy182 <liusy182@hotmail.com>
2019-05-08 22:29:42 +02:00
Jesús Espino
43e6e261d6 MM-14357: Remove store call that can provoke dead locks (#10393)
* MM-14357: Remove store call that can provoke dead locks

* Simplify query

* Adding unit test
2019-03-04 17:52:26 +01:00
Jesús Espino
d937f41233 Idiomatic error handling for app/import*.go (#9473) 2018-09-27 18:55:07 +02:00
Jesús Espino
bba3bbd9f3 MM-11572: Force correct order on messages generated in the bulk (#9244) 2018-08-24 09:53:44 +02:00
Jesús Espino
1f168263a2 Split app/import.go in multiple files (#9179) 2018-08-01 15:45:26 +02:00
Martin Kraft
5872bf9c2f Pr 9039 (#9187)
* MM-11065: Allow to search and get archived channels from the API

* Fixing more tests

* Add some unit tests

* Add includeDeleted parameter to session permissions check function

* More test fixing

* Adding archive channels list in channels search

* Add restriction for archived channel edition

* Reverting permissions checks modification

* Changed the query parameter to include_deleted

* Enable search archive channels as true by default

* Adding tests for verify search on deleted channels

* Allowing to override archive channels during the imports

* Fixed test

* Search in archive channels from the API must be explicitly requested

* Removing includeDeleted parameter from GetChannelByName and GetChannelByNameForTeam

* Back to ViewArchivedChannels config

* Fixing tests

* Reverting GetChannelByName parameter

* Add include deleted parameter on GetChannel functions in plugins api

* Fixing tests
2018-07-30 15:06:08 -04:00
Pradeep Murugesan
ee1d037ca6 Support attachments in post and replies - Bulk import (#9124)
* 9006 - process the attachments of the post

* 9006 enabling the import of attachments in the reply post

* 9006 assert if the post and files are linked

* 9006 fixed the typo
2018-07-20 16:49:49 +08:00
Joram Wilander
5ddb08dcb4 Avoid using app function so bulk loading does not fail when custom emojis are disabled (#9064) 2018-07-12 16:28:04 -04:00
Joram Wilander
6299af0fa5 Add ability to bulk import emoji (#9048)
* Add ability to bulk import emoji

* Improve error handling

* Update test config
2018-07-05 09:08:49 -04:00
George Goldberg
c371ae4db1 Make import compatible with scheme-roles. (#9028)
This doesn't introduce new fields as we would then break backwards
compatability with old "roles" fields. We can add custom fields if/when
we go to "version 2" of the bulk import format.
2018-06-29 17:50:06 +02:00
Jesús Espino
66dc9a5206 Add english missed texts to i18n/en.json (#8987)
* Add English missed texts to i18n/en.json
* Update en.json
2018-06-28 17:12:38 +05:30
Joram Wilander
0d91bf323e Lock bulk importing to master database node (#9012) 2018-06-27 10:08:40 +01:00
George Goldberg
1f65f0e3d6 MM-10233: Add scheme importing to bulk importer. (#8928) 2018-06-21 12:02:35 +01:00
George Goldberg
4ea7f6a2f7 MM-10791: Don't try and generate thumbnails for non-images/svgs. (#8937) 2018-06-12 10:48:45 +01:00
Saturnino Abril
58abb4204e [MM-10645] Remove desktop duration notify props (#8851)
* remove desktop duration notify props

Signed-off-by: Saturnino Abril <saturnino.abril@gmail.com>

* fix test

Signed-off-by: Saturnino Abril <saturnino.abril@gmail.com>
2018-05-25 07:52:17 -07:00
Christopher Speller
686c2fbab7 Structured logging (#8673)
* Implementing structured logging

* Changes to en.json to allow refactor to run.

* Fixing global logger

* Structured logger initalization.

* Add caller.

* Do some log redirection.

* Auto refactor

* Cleaning up l4g reference and removing dependancy.

* Removing junk.

* Copyright headers.

* Fixing tests

* Revert "Changes to en.json to allow refactor to run."

This reverts commit fd8249e99b.

* Fixing some auto refactor strangeness and typo.

* Making keys more human readable.
2018-04-27 12:49:45 -07:00
Jesse Hallam
8491ba5740 Relax 4k post message limit (#8478)
* MM-9661: rename POST_MESSAGE_MAX_RUNES to \0_v1

* MM-9661: s/4000/POST_MESSAGE_MAX_RUNES_V1/ in tests

* MM-9661: introduce POST_MESSAGE_MAX_RUNES_V2

* MM-9661: migrate Postgres Posts.Message column to TEXT from VARCHAR(4000)

This is safe to do in a production instance since the underyling type is
not changing. We explicitly don't do this automatically for MySQL, but
also don't need to since the ORM would have already created a TEXT column
for MySQL in that case.

* MM-9661: emit MaxPostSize in client config

This value remains unconfigurable at this time, but exposes the current
limit to the client. The limit remains at 4k in this commit.

* MM-9661: introduce and use SqlPostStore.GetMaxPostSize

Enforce a byte limitation in the database, and use 1/4 of that value as
the rune count limitation (assuming a worst case UTF-8 representation).

* move maxPostSizeCached, lastPostsCache and lastPostTimeCache out of the global context and onto the SqlPostStore

* address feedback from code review:

* ensure sqlstore unit tests are actually being run
* move global caches into SqlPostStore
* leverage sync.Once to address a race condition

* modify upgrade semantics to match new db semantics

gorp's behaviour on creating columns with a maximum length on Postgres
differs from MySQL:
* Postgres
  * gorp uses TEXT for string columns without a maximum length
  * gorp uses VARCHAR(N) for string columns with a maximum length of N
* MySQL
  * gorp uses TEXT for string columns with a maximum length >= 256
  * gorp uses VARCHAR(N) for string columns with a maximum length of N
  * gorp defaults to a maximum length of 255, implying VARCHAR(255)

So the Message column has been TEXT on MySQL but VARCHAR(4000) on
Postgres. With the new, longer limits of 65535, and without changes to
gorp, the expected behaviour is TEXT on MySQL and VARCHAR(65535) on
Postgres. This commit makes the upgrade semantics match the new database
semantics.

Ideally, we'd revisit the gorp behaviour at a later time.

* allow TestMaxPostSize test cases to actually run in parallel

* default maxPostSizeCached to POST_MESSAGE_MAX_RUNES_V1 in case the once initializer panics

* fix casting error

* MM-9661: skip the schema migration for Postgres

It turns out resizing VARCHAR requires a rewrite in some versions of
Postgres, but migrating VARCHAR to TEXT does not. Given the increasing
complexity, let's defer the migration to the enduser instead.
2018-03-26 17:55:35 -04:00
Jesse Hallam
87762ae62e Improved bulkload error handling (#8491)
* log the config file path used by the server on startup

* return an err if the bulk import command fails

* log the underlying errors that occur when importing

The code assumed all errors meant a missing resource, but it's possible
something else is at fault. Including the error helps pinpoint that more
readily.
2018-03-23 03:33:02 +08:00
Jesús Espino
c158e9a5a0 XYZ-114 Adding mandatory membership to town-square in bulk-import (#8274) 2018-02-15 15:31:48 +00:00
Jesús Espino
6990d052d5 [XYZ-6] Add sampledata platform command (#8027)
* Add fake dependency

* [XYZ-6] Add sampledata platform command

* Creating EMOJI_NAME_MAX_LENGTH as a constant and using it where needed
2018-01-11 10:57:47 -05:00
Chris
816a30397d Role refactor (#7867)
* role refactor

* add missing file

* fix web test
2017-11-21 11:08:32 -08:00
Chris
eb1a00ef5f Reorganize file util functionality (#7848)
* reorganize file util functionality

* fix api test compilation

* fix rebase issue
2017-11-16 16:04:27 -05:00
Joram Wilander
06ec648cf3 PLT-7978 Add websocket event for user role update (#7745)
* Add websocket event for user role update

* Fix tests

* More test fixes
2017-10-31 16:00:21 +00:00
Chris
c5e8cb25ca simplify things (#7735) 2017-10-30 11:57:24 -05:00
Chris
d39a722a81 refactor max channels per team enforcement (#7690) 2017-10-20 20:33:20 -04:00
Chris
8e19ba029f Reduce utils.Cfg references (#7650)
* app.UpdateConfig method

* test fix

* another test fix

* the config override option as-was is just error prone, remove it for now

* derp
2017-10-18 15:36:43 -07:00
George Goldberg
fe41022353 PLT-7883: Fix database race in bulk importer in GetByUsername. (#7611) 2017-10-12 10:00:05 -07:00
George Goldberg
e16bdf8d1d PLT-7789: Fix raciness on Aurora with importing team members. (#7573)
* PLT-7789: Fix raciness on Aurora with importing team members.

* Fix and document return values.
2017-10-04 10:17:58 -07:00
Christopher Speller
29fca51821 Renaming repo 2017-09-06 23:11:59 -07:00
Chris
1adfd0e9be app type transition (#7167) 2017-09-06 17:12:54 -05:00
George Goldberg
e85b5fb988 App: NewLocAppError -> NewAppError (#7327)
* App: NewLocAppError -> NewAppError

* Remove statuscode that got missed.
2017-09-01 11:42:02 -04:00
George Goldberg
50fc6e1e9e PLT-???? Prepare file upload infrastructure for Data Retention. (#7266)
* Prepare file upload infrastructure for Data Retention.

This commit prepares the file upload infrastructure for the data
retention feature that is under construction. Changes are:

* Move file management code to utils to allow access to it from jobs.

* From now on, store all file uploads in a top level folder which is the
  date of the day on which they were uploaded.

This commit is based on Harrison Healey's branch, but updated to work
with the latest master.

* Use NewAppError
2017-08-25 10:38:13 -04:00
George Goldberg
4132a02319 PLT-7183: User/Channel NotifyProps Bulk Import. (#7019) 2017-07-26 08:51:25 +01:00
George Goldberg
02abaabf1f PLT-7123: Add tutorial, flags and favorites to bulk importer. (#7000) 2017-07-21 09:23:27 -04:00
George Goldberg
65289f26e7 Fix some duplicate message IDs in Bulk Importer. (#6850) 2017-07-05 07:49:30 -07:00
George Goldberg
37642a4f1e PLT-6937: Bulk Importing of Direct/Group channels and posts. (#6761)
* PLT-6937: Bulk Importing of Direct/Group channels and posts.

* Show group/direct channels in sidebar.
2017-06-28 07:26:38 -07:00
George Goldberg
a678981861 PLT-6488: Reduce database queries in user bulk import. (#6371) 2017-05-10 09:08:06 -04:00
George Goldberg
6f6b7e4e97 Add Password field to Bulk Importer. (#6268) 2017-04-28 15:01:56 -07:00
George Goldberg
302ec17bee Parallelise Bulk Import. (#6267)
* Parallelise Bulk Import.

* Set worker count through command line flag.
2017-04-28 17:54:04 +01:00
Harrison Healey
fb6f2a123c PLT-5860 Updated copyright date (#6058)
* PLT-5860 Updated copyright date in about modal

* PLT-5860 Updated copyright notice in JSX files

* PLT-5860 Updated copyright notice in go files

* Fixed misc copyright dates

* Fixed component snapshots
2017-04-12 08:27:57 -04:00
George Goldberg
32460bf63b PLT-6198: Use added to channel system message on default channels. (#5969)
Use a different sytem message when a user was added to a default channel
by someone else than when they joined themselves.
2017-04-04 18:19:42 +01:00