Use built-in glib functions to retrieve the list of per-currency price
lists, concatenate them into a single list, instead of doing it all in
hand-rolled loops.
Sorting is preformed by the calling GncTreeViewPrice so this removes
sorting from gnc_pricedb_nth_price.
There's no concurrency concern because gnc_pricedb_nth_price is a
GUI callback and so must run in the GUI thread.
It seems that std::locales created by boost::locale::generator are
not entirely compatible: If used to create a new locale with a facet
for boost::date_time one ends up with the C locale and the facet.
For the time being avoid the problem by using boost::locale to format
dates and times. std::chrono gets calendar functions in C++20 so we
can switch date-time backends once we can adopt it.
We can't use std::locale::global because all streams imbue it by
default and if it's not 'C' (aka std::locale::classic) then we
must imbue all the streams that we don't want localized, and that's
most of them.
Provides error checking for setting the C++ locale from the environment.
This is necessary both because the environment might have an invalid
locale, which would cause an unhandled exception crash.
On windows std::locale("") can't handle some Microsoft-style locale
strings (e.g. Spanish_Spain) so we use boost::locale's gen("") function
to set the locale--though even that can't handle a Microsoft-style
locale string with an appended charset (e.g. Spanish_Spain.1252) and
that's what glibc's setlocale(LC_ALL, NULL) emits.
Testing notes: Based on the averages of 3 runs, the net
user CPU to save the XML file I use is:
10.2 seconds without this change
7.6 seconds with this change
In my environment the first call to the format routine
in question, the call that sets the cache value, is at
the end of the XML load.
The refactoring provides roughly 10% reduction in user CPU
use for XML file load by moving an expensive function
to within an if-clause where the result is used. The diff looks
like a full re-write but only the if statements, indenting,
and commentary changed.
This value is queried on each comparison of split or txn sort function,
which means it is called quite a lot. Avoiding the KVP lookup of this
property gains a lot in terms of CPU cycles.
Because of https://sourceforge.net/p/libdbi-drivers/bugs/24.
This issue causes trouble in save_may_clobber_data() as well, so
work around it by using a SQL query instead of dbi_conn_get_table_list.
Until now GNC_CONFIG_HOME was more or less hard-coded.
Now it can be set via environment variable GNC_CONFIG_HOME.
In addition it will automatically be created to avoid potential
user confusion.
when moving entry between accounts
When using the cut transaction option the 'associated file' value was
not being pasted to the new transaction. Added scheme code to get this
value and save it to new transaction when using 'cut/copy' and then
'paste' operations. When using the duplicate option, a dialogue allows
you to keep the copied association or not. It does not get copied for
autocomplete.
This involves renaming 3 functions:
gnc_uri_get_protocol -> gnc_uri_get_scheme
gnc_uri_is_known_protocol -> gnc_uri_is_known_scheme
gnc_uri_is_file_protocol -> gnc_uri_is_file_scheme
The *_protocol variants are marked as deprecated.
Additionally a number of local variables have been renamed from
protocol to scheme to support this change.
- gnc_uri_get_components will now return NULL as protocol if the input is a normal
file system path instead of a uri (it used to return 'file')
- gnc_uri_get_protocol will now return NULL if the input is a normal
file system path instead of a uri (it used to return 'file')
- gnc_uri_is_file_protocol now returns FALSE if protocol is NULL (it used to return TRUE)
- gnc_uri_is_file_uri now returns FALSE if input is a normal file
system path instead of a uri (it used to return TRUE)
- a new function gnc_uri_targets_local_fs will return TRUE only if its input
is either a file uri or a normal file system path. This function is now mostly
used instead of gnc_uri_is_file_uri in the current code base
- a new function gnc_uri_is_uri is added to check whether its input
is a valid uri (has protocol, path and hostname for non-file uris)
Previously the account color slot has been populated with "Not Set"
when any field for the account has been edited and saved. This routine
should run once and remove all such entries.
There are a very few left that need deeper study, but this gets
rid of most of the noise. For the most part it's just getting rid of
extra variables or removing an assignment that is always
replaced later but before any reads of the variable. A few are
discarded result variables.
It can't, because if b is 0 the function would have
returned already; since b.m_hi is 0 b.m_lo can't be. The assert
reassures clang that this is the case.
We need to compare the magnitudes of the remainder and the denominator
in order to round negative numbers correctly. Note that while gnc_numeric
is constrained to a positive denominator the C++ rounding functions cannot
assume that constraint in all cases.
Combined with the previous commit, this fixes
Bug 796949 - Incorrect conversion of 0,01 USD to EUR
So that the returned price tuple has the two commodities of interest
converted to a common currency. Before the first pair that that shared
any random currency would be returned, perhaps creating an absurd result.
This function complements gnc_time64_get_day_begin/end. There was
time64CanonicalDayTime but this returned noon of the given day, where we
want 10:59am in most cases. I haven't changed time64CanonicalDayTime
directly as that may break assumptions in other parts of the code.
Instead I have created a new function that can be gradually introduced.
Use a variant of xaccParseAmount that allows to ignore the locale's positive_sign character
or the + sign if locale doesn't define a positive_sign character.
In a future redesign it would probably be better to replace use
of xaccParseAmount with some variant of the gnc-expression-parser
but that would require more that a few tweaks to get right.
date-posted to not be saved.
Check the stored GDate for being in the GncDateTime range as well
as the GDate range before returning it and check trans->date-posted
against INT64_MAX instead of 0 before changing it.
The first fix for this bug handled structs tm with ambiguous times.
This one fixes the GncDate constructor when the time is ambiguous
because it's in the DST-change hour, using the same add 3 hours,
construct the LDT, and subtract the 3 hours from the result.
The string constructor handles only simple-offset HH:MM timezones and so
is immune to the bug.
When the preference dialogue is loaded and options are set, the ones
with registered callbacks fire causing parts of Gnucash to be updated.
This was observed with gnc_split_register_load being executed 5 times
for each open register when the preference dialogue was loaded.
To overcome this, a couple of functions have been created to block and
unblock all registered prefs and used while the preference dialogue is
loaded.
And don't ask to save a not-dirty or empty book, fixing
Bug 794870 - If no book is opened, gnucash still asks if the user wants
to save changes when opening a file
These are queried continuously by the owner tree view (on Customer/Vendor/Employee
Overview pages) and recalculating them is an expensive operation.
The cache will be invalidated each time a lot reated to the owner
changes (modify or delete). The net effect is a huge responsiveness
improvement of said overviews in case of a large book.
Any operation that can overflow will throw an underflow if it's a
negative number. The C interface needs to catch both to prevent
unhandled exception crashes,
That is let the percentage increase gradually. The current granularity is still
very rough, but at least it gives an indication of getting closer to fully
loading the data. The previous configuration on the other hand only suggested
something was happening but with no indication where in the load process
gnucash was.
This prevents calling xaccAccountRecomputeBalanceInCurrency on each split that gets added,
which was exponentially increasing load times. On a huge test book the
load time dropped from 53 minutes to 1m20s.
The core issue was that the delete visitor was never called because its parameter
type (char *) didn't match the boost::variant type (const char *).
Fixing the visitor's parameter type also require a const_cast
back to char * because that's what g_free takes as argument.
The rest of this commit is merely fixing KvpValue instantiations that
tried to create a char* KvpValue from a stack based const string instead
of a heap allocated one. That would bomb out on calling the
delete visitor.
Splits were not marked for deletion if the transaction is read-only
and the account is not marked for deletion yet. The net result is
that split will not be freed later on.
However xaccSplitDestroy is also called from a Transaction's do_destroy.
At that point accounts are not necessarily marked for deletion yet (like
is the case when a datafile is closed). This turned out to be a problem
for invoice post transactions (which are also read only) and hence
would cause memory to leak.
Always use absolute paths for configured directories (BINDIR etc.)
Abstract out the guts of gnc_gbr_find_foo_dir for foo in lib, bin, and data.
etc requires special handling because of the way it's treated if prefix
begins with /opt.
Always fall back on the configured directory if binreloc is disabled and
no default is passed in.
Casting a char* to a struct containing a uint32_t is not universally safe
due to alignment constraints on reads on some platforms. Copy our possibly
unaligned source data into an aligned area of memory to avoid SIGBUS on
armhf.
Reported by vorlonofportland in PR#403. This commit the John's optimized
version of Vorlon's proposed fix.
- add -Wno-deprecated-declarations to CXX_FLAGS as well. This was
reported by vorlonofportland in PR#401 to become necessary for glib 2.58
as that has deprecated g_type_class_add_private which appears in our
c++ code.
- change -Wno-deprecated-register into -Wregister. The former appeared to
be a clang dialect and alias for the latter (see
https://github.com/Barro/compiler-warnings for an overview of clang
and gcc warnings). It was moved to global CXX_FLAGS as it can only be
added for g++.
The KVP value for the qof_book_get_num_days_autoreadonly was being
called many times so it makes sense to cache it in the book to avoid
the KVP lookup.
Each was used exactly once and simply wrapped a function call.
Also replace static function time_parse_failure that just returned a
constant with the constant.
This report had renamed "Action" to "Charge Type". For consistency,
let's rename it back, and add migration path in options.scm for
saved-options. 2 fewer strings for translation.
test_suite_gncInvoice sets up the test suite. It's not part of the test
runtime, so stack variables in it have gone out of scope by the time the
tests are actually run. Making invoiceData static makes it permanent so
it exists at runtime.
With the options dialogue open when Gnucash is closed this error is
logged in the trace file, gnc_close_gui_component() component not found.
This is fixed by reversing the components list found for session so the
options dialogue is closed before the report window.
Anticipating that some users might prefer to see exact prices,
add a preference to General>Numbers to configure whether prices
are rounded to decimals or are displayed as exact fractions.
When printing numbers convert them to a new decimal denominator with
rounding if the passed-in print info specifies that they should be
forced and rounded.
Make the default price settings forced and rounded.
Pass the price currency to gnc_default_price_print_info and
use the currency's fraction * 100 to determine the round-to
denominator and the number of decimal places to display.
First, save isn't necessary if the book is dirty, so don't... but that
means that the book has to be marked dirty after a session swap. No more
laziness.
Second, regardless of the outcome of inner_main_add_price_quotes the
session must be destroyed to remove the lock.
A couple of cleanups in QofSessionImpl::save as well: Rewrote the
descriptive comment to reflect how it really works when the backend has
gotten disconnected and removed the superfluous qof_book_set_backend
with the backend that we'd *just gotten from the book*.
Change all instances of bugzilla.gnome.org to bugs.gnucash.org, reflecting
our migration to a self-hosted bug tracker.
Inform the Translation Project Coordinator at release that this affects
translatable strings and that all message catalogs have been updated.
I think this crash is triggered because the 'account' variable
defaults to the first available AR account. If there's no AR account
it becomes null, and querying null's default book leads to segfault.
I guess I can fix segfault too by fixing gnc_account_get_book.
In addition to not begining to edit already-loaded transactions,
don't try to load splits that are already loaded. It shouldn't
be possible to load a transaction without also loading its splits.
Change some plain string literals to std::string constants, which helps
avoiding typos and also saves some string constructors/destructors
in the KVP lookup. Nevertheless the functions in Account.cpp do not
contribute that much to the overall UI speed, but whatever.
The underlying problem was that the vendor object remained in infant state
That confused the backend code so it used an sql INSERT statement instead
of an UPDATE statement to write back the changes. As the object already
existed in the db this would fail.
The fix is to ensure the object doesn't remain in infant state during
sql loading. See the bug report for a more detailed explanation.
The function qof_book_use_split_action_for_num_field gets called quite a
lot in each register display refresh (due to sorting all splits from
Split.x's xaccSplitOrder function), but it always used to use a KVP
lookup, which is rather expensive compared to accessing a gboolean member
variable.
To get rid of this cost, I had to remove the KVP lookup in this
simple-looking function. The pattern is this: A gboolean cache variable is
introduced, along with an isvalid flag. The lookup makes the expensive
KVP lookup once, then caches the value. The GObject property mechanism
offers a callback for when the setter was called, which is used to mark
the cached value as invalid. A parallel setter method (here:
qof_book_set_option) also just marks the cache as invalid. This covers
all setters, and the getters will use the cached value except for their
first invocation.
The NUM_FIELD_SOURCE feature was introduced in 2012 by the very large
commit 7cdd7372 and apparently its costs never were a problem
until the KVP lookup became more costly due to the std::vector
construction and destruction.
Turns out that the on-the-fly conversion from const char* (the KVP_OPTION_PATH
constants) to std::string with their immediate deletion afterwards is
a quite costly operation. Avoiding this is surprisingly easy: Just keep
local std::string objects at hand, and they don't have to be created
and deleted anymore.
The more optimized solution might be to turn the std::vector<std::string>
into a std::vector<GQuark>, but this commit at least improves the picture for now.
An extra XaccTransBeginEdit, never committed, for transactions that
the backend tried to load when they were already there. That made
the register think that something else had it open.
Instead of reporting an error and declining to load the file (XML)
or failing to enter a value (SQL) when a bad date is found in the
database, use a 0 time stamp (1970-01-01 00:00:00 UTC). Adds a warning
in SQL backends; there was one already in XML.
1. Don't even check for price/exchange rate on template transactions,
there's no point.
2. Extract function get_transaction_currency:
a. Check all split commodities are valid, abort transaction creation if
not.
b. If the template transaction's currency isn't used by any of the
splits set the new transaction's currency to the first-found currency if
there is one, otherwise to the first-found commodity.
3. Fix a minor typo in a comment.
Limit the range of the random value to 1..1000 to prevent overflows,
particularly in number-of-periods or number-of-years variables.
While we're at it, g_random_int and g_random_int_range return ints so
piping the result through gnc_double_to_numeric() doesn't make much
sense. That's removed, we just construct a gnc_numeric.
Provide a backup recovery function that instead of dropping primaries
and restoring backups merges the primaries and backups. This should
handle a worst-case safe-save failure where the backup tables don't
have a complete set of rows for some reason.
We don't use floats in GnuCash, we use doubles (and those as little as
possible), but dbd-sqlite3 is broken in that it stores only floats.
Simply casting floats to doubles introduces bogus additional digits
that can cause round-trip tests to fail. Instead convert floats to
doubles by multiplying by 10E6, rounding, then dividing by 10E6.
Removes a public function, GncDbiSqlConnection::table_manage_backup that
should have been private in the first place.
Better encapsulates table renames and drops with private functions and
handles cases where there exist some primary tables and some backup tables.
SQLite backend.
Release Note: This bug caused data loss if you saved your SQLite3
database to a different file or database.
The problem is that in SQLite3 (though not in MySQL or PgSQL) the
subquery ((SELECT DISTINCT guid FROM transactions)) (note the double
parentheses) returns only the first guid in the subquery's results.
Some transactions are loaded by special queries and those queries are
also used to retrieve the transaction's slots so they weren't affected.
First change is to ensure gncEntry rounding is consistent. Internally
calculated values in the entry are never rounded. Consumers of
gncEntry's calculated values can request them either rounded or not.
Next use a pragmatical approach for calculating values on invoices based on
the entry values: do the rounding such that we never
create an unbalanced transaction while posting
That means
- round each entry's net value before summing them in net total
- accumulate all tax totals on invoice level per tax account before rounding
and round before before summing them in a global tax total
Hopefully this will catch a few more rounding issues in this area.
A complete solution can only offered if we allow users to manually correct
tax entries. This requires changes to user interface and data format
so that's not going to happen in gnucash 3.x.
strptime/strftime supports various modifiers to their parameters.
'E' and 'O': alternate locale-specific formats
(used in default format for Persian, Oriya, Azerbaijani)
'-': padding
(used in default format for Czech)
GnuCash passes dates as integer y/m/d without using locale-specific
formats, so we need to strip out 'E' and 'O' from the format when
scanning dates or determining separators in gnc-date.
None of '-', 'E', or 'O' are supported by boost (and '-' causes
errors), so strip them out from formatters in gnc-datetime as well.
See https://bugzilla.gnome.org/show_bug.cgi?id=795247.
This will probably need more refinement because the multiplications
are still missing rounding methods, but the changes in this commit
will already allow guile code to correctly create entries.
Clearer syntax helped find flawed test - while set-tm:mday directly
accepts 1-31, set-tm:mon accepts 0-11 to represent 1-12, therefore
must minus 1. set-tm:year accepts 92 to represent 1992, therefore must
minus 1900.
We want to sanitize render-options-changed, therefore it must return
an html-object. Unfortunately this is not accessible to
app-utils/options.scm. If we move this function to
report-system/html-utilities.scm, it can access html-objects.
Also rename it to gnc:html-render-options-changed
The swig 3.0 generated python wrappers trigger a warning converted into an error issued
by gcc 8.0 for using strncpy as follows:
strncpy(buff, "swig_ptr: ", 10);
The reason is this call will truncate the trailing null byte from the string.
This appears to have been fixed in swig master already but that's not released yet
so let disable the warning when compiling the swig wrappers until it is.
This commit contains another round of cleanups in the
timespec to time64 conversion. There were a number of
false assumptions that time64 = 0 would be a bad date
in the xml parser. This commit corrects enough of them to
eliminate the bug. Further cleanup is probably advised but
can be done at a later stage.
Also Bug 791825 - Accounting Period dates off by 1.
The DST start/end dates were reversed *and* the DST offset had the wrong
sign in Windows, resulting in the effective timezone always being one to
the west off (i.e. PDT was -9 and PST was -8).
Take 3: Catch encoding exceptions from trying to read a string into
Scheme using scm_from_utf8_string and try again using
scm_from_locale_string. If that throws too, give up and log a
warning.
First, remove the unnecessary locale push & pop on <CT_TIME64>load.
Second, the registry accesses were caused by using g_win32_get_locale
to convert the Microsoft locale strings to POSIX ones. We don't care
what kind of string we get as long as we can pass it back to setlocale,
so remove that.
Third, gnc_push/pop_locale were used only in backend/dbi in a
very limited way and did much more than was necessary, so
convert them to C++ inlines in gnc-backend-dbi.hpp that does
only what we need them to.
properly - corrupted business data
Turned out to be a pointer/value mismatch between <CT_NUMERIC>load()
and most of the setter functions, so the address was getting set
as the value.
This crash started to appear as of commit 80dbb9940b because the sequence
of split loading has changed as a result of the query optimizations.
Invoice transactions get loaded before the general transaction loading happens.
However because of this, when an invoice transaction was encountered again
during general transaction loading, it was (correctly) not created again
AND (incorrectly) not opened for subsequent editing. This caused
an assert to fail when the splits for this transaction are loaded
shortly afterwards. The solution is simply to ensure all transactions
are opened for editing during the general transaction loading call.
set/get_locale are apparently very expensive on Mingw64, and setting
the C locale for extracting a string is unnecessary.
Unfortunately the released version of libdbi still uses strtod so
setting the C locale *is* still necessary for retrieving floats
and doubles and for passing queries.
Thanks to Mkubat for the diagnosis.
The mingw-w64 toolchain bizarrely substitutes scm_to_locale_string()
for scm_to_utf8_string(). This results in latin1 (yeah, "locale" is
a lie) instead of utf8 which causes an assertion in
g_utf8_collate_key().
Perhaps equally bizarre, the compiler doesn't make the substitution
with scm_to_utf8_stringn(), so use that instead.
Bills and invoices that are posted and subsequently unposted again still store their
posted account internally as a convenience to the user (upon reposting the old
account will be offered by default) so it's not a reliable test for the posted state.
The posted transaction on the other hand is guaranteed to only exist when the invoice is
posted. This should fix a slew of small and perhaps larger side effects, such as
a posted bill still appearing as editable, critical warnings when creating new bills/invoices
and so on.
An odd corner case: BST apparently came off of DST at 23:00 26 Oct 2014,
so midnight that day was ambiguous about being DST or not; that causes
the local_date_time constructor to throw in spite of the tm.is_dst element
being 0 (meaning pick standard time).
Instead of just failing in that case, try constructing a local_date_time
three hours later then adjust it back three hours. If *that* doesn't work
then throw a std::invalid argument.
After much thrashing this turned out to be caused by a date string
with a 3-digit year and that caused an unexpected boost::bad_cast
exception from boost::posix_time::time_from_string().
To prevent that and anything like it, pre-parse the string with
regular expressions to classify them and split out the timezone
if there is one. If neither (perhaps eventually none) of the
regexes match throw std::invalid_argument. The C function will
catch this and return 0.
91f4b19 changed the test for gncInvoiceDateExists from date != 0
to date != INT64_MAX, which isn't backwards compatible, so test
for both. But the submitted file had a posted date of -1 so
gncInvoiceIsPosted returned true anyway. That's not consistent with
the logic in dialog-invoice.c, which checked gncInvoicePostedAcc != NULL,
a better test. The result was that the "Post" button lit up but
gncInvoicePost returned immediately, doing nothing, so change
gncInvoiceIsPosted to use gncInvoicePostedAcc instead.
Be smarter about what is path and key for each slot.
Instead of assuming a slash is always a path separator (first attempt
on unstable) or never a separator (second attempt),
track the parent path while loading kvp slots from the db
and deduce the slot's name by substracting this parent path.
