gnucash/libgnucash/doc/books.txt

/** \page bookperiods Books / Accounting Periods

API: \ref Book\n

\section periodsintro Implementation Overview

		   Linas Vepstas <linas@linas.org> December 2001
                 Last Updated August 2003

A top, unimplemented request for GnuCash is the ability to 'close 
the books', that is, to add support for 'accounting periods'.
Partial support for books has been added to the GnuCash engine; 
and a GUI is partially finished.  This file reviews the 
implementation design choices and the current design status.

\subsection periodsdefines Definition

An accounting period or 'book' is a set of accounts and transactions
that, once closed, must never be modified again.   Books are typically
closed once a quarter, or once a year.  Generating a report from a 
closed book tells you how well you did for that year (or quarter).
The provision against modifying a closed book helps ensure correctness,
and discourages cheating.   Note that the closing balances of a closed
book must be carried forward as the opening balance in the next book.
(But only for asset, liability and equity accounts. Opening balances
are zeroed for income and expense accounts.)

Note that the correct handling of depreciation, capital gains and
other similar accounting concepts requires 'Lots' to be correctly
handled.  Lots are a way of identifying that an item bought in one
transaction is the same as that sold in another transaction. 
When a book is closed, the entire lot must be brought forward,
and not just the account balance, because depreciation, capital
gains, taxes, etc. depend on the dates of the originating
transaction.  See 'lots.txt' for details.

API: \ref Lot

\subsection periodsrequired Requirements

Must have good performance (large data files usually mean poor performance).
Use the idea of 'books' to prevent file bloat.  Must have access to 
historical data.  Must be able to create bar-charts, graphs, reports
of multi-year data (i.e. create reports spanning multiple books).

\ref Period

\subsection periodsstatus Status

The top-level structure that holds references to all of the data in
a book is implemented in src/engine/qofbook.c.  The routines to 
split one book into two, automatically setting account balances, 
etc. are implemented in src/engine/Period.c.  The actual 
implementation used is "Plan A" from the list of choices 
below.  The actual implementation is not yet complete, see 
"Implementation Notes" at bottom for the current status.

\section periodssolution Possible Solutions

API: \ref Query

Listed in order from worst to best:

\subsection periodsdelete Plan F:

Simply 'delete' old transactions, and adjust the equity to make up for 
this.  More specifically: Split one file into two, with only 'old' 
transactions in one, and only 'new' transactions in the other.

I believe that this can be 'easily' coded by creating a second instance
of a qofbook structure in memory, copying all the account info 
into it, and using Query.c to copy in only the 'old' transactions.
(and v.v. using Query.c to delete the old transactions in the other copy.)
Then, look up the ending balance on all asset/liability accounts
in the 'old' book, and create new transactions in the 'new' book
that transfers that balance amount to an equity account.
The transfer description is, of course, 'opening balance'.
Balances of income/expense accounts are zeroed out.

I believe this code would be easy to write in C or scheme. There may be
a few bugs/difficulties lurking in qofbook that might trip things up.
Also, at a minimum, there needs to be a GUI dialog, asking for the 
date on which to close the books.

(A fancy but optional GUI dialog/wizard might ask 'which equity 
account to transfer the opening balances, and what the description 
should say. This GUI is optional, since, after all, these can be 
tweaked by hand, and its only done once a year or once a quarter.)

(An even fancier GUI would remember how often the books should close:
1,2,3,4 times a year, 12 times a year, whatever, and 'remind' you 
when that happens.)

(Another 'fancy' feature might be to not allow user to close book until
all 'old' transactions have been cleared/reconciled.  But that might be 
a bit much for non-bank accounts).

API: \ref Transaction

\subsection periodsf Pros & Cons of plan F:

- pro: simple. The simplest option.
- pro: truncated file loads much faster
- pro: old/irrelevant accounts can be safely deleted from newest file
     (while still being preserved in old files).

- con: impossible to generate 5 year reports, multi-year graphs.  This
     would really hurt, esp, when tracking stocks/mutual funds/retirement
     accounts over a number of years.

I think this last one is the Achilles heel, the torpedo in the rudder
that sinks the boat.

\subsection periodspland Plan D

As above, but instead of deleting, add a kvp to each transaction stating
'/book/closed-on=12.31.2000'.  Then modify the default query for the 
registers so that the only displayed transactions are those that are *not*
part of a closed book.  Modify the Query GUI dialog to add 'book' as 
a query parameter. 

- pro: easy access to historical record

- con: slow loads; file size not reduced.
- con: dealing with opening balances, equity, is icky.
- con: can't delete/hide old/stale accounts.

We move on....

\subsection periodsplanc Plan C

As in plan F, but instead of creating two books, clone the account tree
into two: 'old' and 'new'.   The old and new accounts are identical, 
except that they get different guid's.  Every  account in the old 
tree gets a kvp in it: '/book/closed-on=12.31.2000'.   We don't copy
or delete any transactions; instead, we reclassify them: Old transactions 
are transfers between old accounts, new transactions are transfers 
between new accounts. 

The account summary needs to be modified to show only 'new' accounts 
by default.  The transfer-from pop-down needs to be modified to show 
only 'new' accounts only, and never the old accounts.

Transfers between closed and open accounts are never allowed (this is 
validated/forced in the engine).  Opening balances are handled just as 
in plan 'F'.   User can only view data in closed books, and not change 
it.  

If we allow books to be re-opened, then the 'starting balance' equity 
transfers must be deleted.   We can save 're-opening' for some future 
day.

The 'starting balance equity transfers' must have a kvp pair in them: 
'/book/closing-balance-of-account-guid=0xdeadbeef'.
This way, we know that this transaction is associated with the closure
of the book on some specific account, and that way, we can find this
transaction someday in the future, if we ever need to.

Each new account needs to point back at the copy that is its 'old' self. 
(these don't have to be C pointers, they could be some suitably clever
kvp: '/book/previous-guid=0xdeadbeef')  This continuity is needed in order
to be able to create reports that scan over multiple books.  The Query.c
interface needs to be modified so that it searches only new accounts,
or it searches new accounts *and* their corresponding 'old' copies.

(There are three ways to deal with this account continuity issue:
- don't deal with it in query.c:  force various GUI dialogs to explicitly 
    formulate queries involving the /book/previous-guid string.  
    but this gets messy in the GUI's.  May lead to excess cut-n-paste
    of similar code between different GUI's.
    
- 'hide' the distinction between 'old' and 'new' in query.c: 
    the users of query.c need only to specify a boolean flag: search 
    closed books: yes/no.  However, this is conceptually ugly, and 
    prevents query from doing low-level queries on specific 
    books.
    
- create query utility wrapper/pre-processor that takes a query,
    and then modifies it to search through closed books as well.
    This is the 'cleanest' solution. ??
    
- All these are delicate, and need a little more thought and 
    exploration. Goal is to simplify queries, not burden the system 
    with cryptic, complex code.
)

I believe that if we can deal with the account-continuity issue in query.c
or in a wrapper thereto, that there are no remaining issues with
reporting.  i.e., for any given report, we are either reporting
data in closed books, or not.  Different reports should have different
defaults.  e.g. income/expense pie chart never looks at old books. 
asset-value-over-time-bar-chart always looks at closed books.


- pro: safer than plan F, since we really can enforce the 'you aren't 
     allowed to edit closed books' rule.
- pro: solves the old-account/new-account problem, since new accounts 
     can be edited/deleted without damaging old account.
- pro: solves the historical reporting problem.

- con: queries are potentially slow, loading of file is potentially slow.

But now we have enough info to propose the final solution:

\subsection periodsplana Plan A:

The kvp markup of plan C coupled to the multi-file solution of plan F.
In initial startup of GnuCash, only the 'current' book is loaded.  
If user asks for a report that requires data from old books, then
we have to pause to load one or more of the older books.   

If the books are stored as separate files, then the 'current' book 
needs to somehow know the filenames of the old books.  I recommend
against storing the books as different sections of one file, for 
many reasons:
- risk of corruption of old books 
- bloated file size 
- the single-file solution would need to invent a 'directory' so 
   that the location of the old books in the file can be quickly 
   found and lseek()'ed or mmap()'ed.  But why invent a directory? 
   Unix already provides directories!

I recommend that every book get a unique guid.  The current book
would know the guid's if its closed book progeny.  The filename
would incorporate some compressed version of the guid (and/or
the date of closure).

Optional:
every book gets not only a unique guid, and also stores some 
meta-information (as book-level kvp's):

/book/title=some-user-supplied-name\n
/book/notes=user-supplied-descriptive-comments\n
/book/start-date=xxx\n
/book/end-date=xxx\n
/book/previous-book-guids=(list 0xa 0xb 0xc)\n
/book/accounting-period=enum {none, week, month, quarter, trimester, year}

\subsection periodsaprocon Pro's & Con's

I am not aware of any con's to plan A at this point.

\section periodsoverview Implementation Overview

Plan A has been implemented in the engine.  To quickly summarize:
- Partitioning involves splitting one book into two, called the
   "old, closing book", and the "current open book".

- Accounts are copied between old and new books.  One of the copies
   is issued new GUID's, but the rest of the account data is copied.
   KVP pairs are then added so that each copy points at the other,
   and can thus be easily found.  The "gemini" KVP keyword is used.
   The opening balance is zeroed for income accounts.  A transaction
   is created to set the correct opening balance on asset accounts. 
   The transaction is a transfer from equity.  An equity account is
   created automagically, if needed.

- Transactions. Transactions are partitioned, and end up either
   in the old or the new book.  Splits move with transactions.
   Note that some transactions, associated with open lots, may be
   kept in the new book (See below).  

- Lots.  If a transaction has a split in an open lot, then that 
   transaction is not moved to the closed book.  It needs to stay 
   with the open book.  If a lot is closed, and all of the other 
   lots associated with all of the transactions in this lot are 
   also closed, then the lot may be moved to the closed book 
   (and all of the other associated lots must then also be moved).

- Prices. Prices are sorted into the new and old books according
   to the date on the price.

- Scheduled transactions/recurring transactions.  These are left
   in the new book, untouched.   They are not copied into the old
   book, and no trace of their existence is left in the old book.

- Business Objects. Not implemented.

\section periodsnotes Implementation Notes

- src/engine/Period.[ch]
      Implements the main logic to split one book into two, and populate
      it with the appropriate keys, markup, etc. and to carry balances 
      forward, etc.

-  src/engine/gnc-lot.[ch]
      Implements accounting Lots.

-  src/engine/Scrub2.[ch]
      Implements simple FIFO for lots.  Data is scrubbed with respect to
      this FIFO before the books are closed.  Commodity accounts (e.g.
      stock accounts) must have a coherent Lots structure before books
      can be closed.  The scrubber makes sure there's nothing hanging
      out.

-  src/gnome/druid-acct-period.[ch] 
      Implements a druid interface to allow user to specify book closing 
      dates, add a title and notes to a book, and walk through the process.
      Uses FreqSpec.[ch] and the widget in gnc-frequency.[ch] to allow
      the user to specify the frequency of book closings.

- The XML-file backend can store multiple books in one file.  There 
   is currently minimal support for writing out multiple books, 
   one per file (this enables faster load, by not loading old books).  
   There is currently no support for loading multiple books from multiple
   files.
   
- (Details about the dbi backend are missing.)

\section periodsannounce ANNOUNCE: Book Closing Beta 2

Books AKA Accounting Periods can now be closed by going to the 'Action'
menu on the main window, and selecting 'close books'.   This will popup
a druid that will allow you to select closing dates.  Books are closed
one at a time: after each one, you can go back and alter the close date.  
A book is 'closed' by moving transactions and prices earlier than the 
indicated date to a separate file.

As a side-effect of book closing, capital gains (losses) will be
automatically computed using a FIFO accounting method.  Stocks 
or other non-cash assets that haven't been sold will remain in the 
currently open book, even if the purchasing transactions were made 
a long time ago.

The biggest drawback to the automatic computation of the gains/losses
is that there is no GUI to specify alternate accounting methods
(e.g. LIFO, or hand-picked lots, etc.)  There is basically no practical
way to 'undo' the results of the gain/loss computations.

The other main drawback to book closing is that this will prevent
multi-year (multi-period) reports & graphs from being generated.
The old data will be in separate files, and there is currently no
way to load several files at once.

\section periodsissues Open Issues/Questions that Need Discussion:

- How to handle business objects? e.g. vendors, customers should be
   copied into both old and new books.  But invoices should be in one 
   or the other.  Need to document which is which.
   Copy:
   customer -- ok, has guid, has for-each
   employee\n
   No-op:
   address -- do nothing

- Discussion Q: What should the naming convention be for the different 
   books?  When the XML file backend is used, different books need to be 
   stored as different files (in order to avoid the performance 
   penalty of a large file load).  Currently, what's implemented is\n
   book-1dc750aa3e6fd045c13ac8afb1a9ac03-my-gnucash-file.xac.gml\n
   where the number is the GUID of the closed book (needed to be able to
   quickly find the book/file) and 'my-gnucash-file.xac' is the name of
   the original file whose books were closed.\n
   Need to change the name to include the word "archive".

- Discussion Q: When saving books, make the book title part of the 
   book name (user convenience).

- Should closed books be allowed to have unreconciled transactions?
   Answer: probably.  Should there be a warning? 

\section periodstodo Open Issues / ToDo

- Change GUI to allow user to specify a default equity account
   for dealing with opening balances.  Should be done with kvp markup. 

- Fix crash when exiting gnucash after closing books. 

- The filename of the old, closed books should probably be saved 
   in the KVP tree of the current open book.  This need be done only
   from the file backend.

- Need to mark closed book as unalterable, and respect that markup.
   I think there's a 'closed' flag in the book, but I don't think its
   respected.

- The book closing GUI (druid-acct-period.c) needs to save/restore 
   period end date (the FreqSpec) to KVP on open book.  This would be
   easy once we have a freq-spec-to-kvp and freq-spec-from-kvp routines.

- Handling of lots in book closing is implemented but is poorly tested.
   Need to write test cases.  Also test cases for prices in book
   closing.

- price-saving in the SQL backend is probably borken, its certainly
   untested.  Need to remove old deprecated price-lookup mechanism, 
   and replace w/ qofquery.

- Need to provide for loading of closed books, because this is needed
   for reports.

- Handling of multi-book reports ???  Need to work out the recommended way
   of making this happen....

- Have some way of remembering the quickfill text from older books.

- Possibly neat idea: 
   Book closing, as currently implemented in the GUI, is driven entirely
   by the date-posted.  There is no (planned) interface to allow you to
   exclude some certain transactions from a particular closing (although
   it would be 'easy' to add this: right before I close a book, I have a
   list of transactions which can be added to/removed from).

- add 13-period support to FreqSpec and the FreqSpec widget.  
   e.g. from the mailing list:
   One of the calendars my company uses is like this:
   13 periods
   1st period begins jan 1st, partial first week plus 4 weeks.
   2nd - 13th period begins on a sunday every four weeks.
   13th period - 4th week may be less than full week because it ends on 12/31.\n
   For 2003:\n
   01. 01/01 - 02/01\n
   02. 02/02 - 03/01\n
   03. 03/02 - 03/29\n
   04. 03/30 - 04/26\n
   05. 04/27 - 05/24\n
   06. 05/25 - 06/21\n
   07. 06/22 - 07/19\n
   08. 07/20 - 08/16\n
   09. 08/17 - 09/13\n
   10. 09/14 - 10/11\n
   11. 10/12 - 11/08\n
   12. 11/09 - 12/06\n
   13. 12/07 - 12/31\n
                                                                                
*/
=========================== end of file ========================