Блогчетање

Данилово блогче

Wed, 23 Nov 2011

Doing 'git clone' over either of git:// or http:// backed by git-http-backend results in big memory usage for big repositories (mostly in 'compressing objects' phase). This means that you can usually kill any git server hosting big repos by doing concurrent 'git clone' runs of those repos at the same time.

Instead of doing that, I'd like to keep benefits of the git-http-backend for all users except those using 'git clone'. Anyone has any ideas on how to do that?

OTOH, maybe I am looking for the wrong solution. git supposedly does mmap() of the pack files, and the only thing I need to do is ensure packs are suitably small so they could me "swapped-out" (which equals to being discarded from memory with mmaped files). Or, since git does a mmap() of uncompressed temporary file, maybe I can get git to store uncompressed data instead, allowing it to mmap them directly?

I wouldn't be surprised if I am entirely on the wrong track here. If anybody has ideas on what to do to run a git server with big repositories without needing gazilions of memory, please direct me in the comments section below.

[18:21] | [] | # | G | | TB
In theory, a git clone should just stream the entire pack file from the server to the client, with minimal processing, and definitely not a pile of memory usage (though it may look that way due to the use of mmap).  It might help if you did a "git repack -a -d -f" on the server to regenerate the pack file.

If you experience excessive non-mmap memory usage from "git clone", I'd suggest asking about it on the git mailing list, and checking if your experiences match upstream's expectations.
— Posted by Anonymous at Wed Nov 23 18:39:47 2011
The favorable and probably easiest approach is git bundle [1]. You create a bundle, put them somewhere people can download them separately then fetch on top to get latest changes.

[1] http://thread.gmane.org/gmane.comp.version-control.git/184490/focus=184497
— Posted by pclouds at Thu Nov 24 14:14:42 2011

Name:


E-mail:


URL:


Comment:


Contact
Danilo Segan

This is blog (web log) of Danilo Šegan (or Данило Шеган).

Archives
2017-Jan
2013-Dec
2011-Nov
2011-Oct
2011-Aug
2011-Jul
2011-Jun
2011-May
2010-Oct
2010-Aug
2010-Jul
2010-Apr
2010-Mar
2010-Feb
2010-Jan
2009-Dec
2009-Oct
2009-Aug
2009-Jun
2008-Oct
2008-Aug
2008-Jul
2008-Jun
2008-May
2008-Apr
2008-Mar
2008-Feb
2007-Dec
2007-Oct
2007-Aug
2007-Jul
2007-May
2007-Apr
2007-Feb
2007-Jan
2006-Nov
2006-Oct
2006-Aug
2006-Jul
2006-Apr
2006-Mar
2006-Feb
2006-Jan
2005-Sep
2005-Jun
2005-May
2005-Apr
2005-Mar
2005-Feb
2004-Dec
2004-Nov
2004-Oct
2004-Sep
2004-Aug
2004-Jul
2004-May
2004-Apr
2004-Mar
2004-Feb
2004-Jan
2003-Dec
2003-Nov
2003-Oct
2003-Sep
1983-Mar

< November 2011 >
MoTuWeThFrSaSu
  1 2 3 4 5 6
7 8 910111213
14151617181920
21222324252627
282930    
Categories

Links
Kvota.net
Prevod.org
My study page
Srpski.org
GNOME
Friends' Blogs
alex (en)
bc (en)
Bojan Živanović (sr)
Carlos (en)
Goran (sr)
imp (sr)
lilit (sr)
Oskuro (en)
Zombie (sr/en)
Feeds
RSS