Reduce memory consumption of initial cache population

[Lewis Baker]

The current cache syncing logic attempts to pre-fetch all change information into memory using batched Perforce commands before inserting new records into the SQL database.

Whilst this method improves performance of incremental updates to the cache it tends to consume large amounts of memory when performing the initial cache population of a large repository (>20K changes) which can swamp the web server machine making the initial cache population to take longer.

It would be better to use some form of incremental batch processing, clearing cached data in-between batches, to reduce the total amount of memory required to perform the initial cache population.

[Lewis Baker]

Ticket #639 is a duplicate of this one (patch included).

[Lewis Baker]

I've updated the plugin to perform cache syncing in batches, retrieving information for 1000 changes at a time into memory and committing to the Trac database before purging the in-memory cache.

See ​version 0.4.2.

Some quick tests on our repository showed a significant decrease in virtual memory usage (from >1GB to <100MB) and time taken (from >1.5 hours to <25 mins).

Thanks to r.blum@… for the initial patch.