Ticket #630 (closed enhancement: fixed)

Opened 2 years ago

Last modified 1 year ago

Reduce memory consumption of initial cache population

Reported by: lewisbaker Assigned to: lewisbaker
Priority: high Component: PerforcePlugin
Severity: normal Keywords:
Cc: Trac Release: 0.10

Description

The current cache syncing logic attempts to pre-fetch all change information into memory using batched Perforce commands before inserting new records into the SQL database.

Whilst this method improves performance of incremental updates to the cache it tends to consume large amounts of memory when performing the initial cache population of a large repository (>20K changes) which can swamp the web server machine making the initial cache population to take longer.

It would be better to use some form of incremental batch processing, clearing cached data in-between batches, to reduce the total amount of memory required to perform the initial cache population.

Attachments

Change History

08/22/06 18:59:04 changed by lewisbaker

Ticket #639 is a duplicate of this one (patch included).

08/22/06 22:50:12 changed by lewisbaker

  • status changed from new to closed.
  • resolution set to fixed.

I've updated the plugin to perform cache syncing in batches, retrieving information for 1000 changes at a time into memory and committing to the Trac database before purging the in-memory cache.

See version 0.4.2.

Some quick tests on our repository showed a significant decrease in virtual memory usage (from >1GB to <100MB) and time taken (from >1.5 hours to <25 mins).

Thanks to r.blum@gmx.net for the initial patch.


Add/Change #630 (Reduce memory consumption of initial cache population)




Change Properties
Action