Github messages for mblaze
 help / color / mirror / Atom feed
* [PR PATCH] mthread: reduce memory usage
@ 2022-03-13 19:35 julianrother
  2022-03-13 19:42 ` [PR PATCH] [Closed]: " leahneukirchen
  2022-03-13 19:42 ` leahneukirchen
  0 siblings, 2 replies; 3+ messages in thread
From: julianrother @ 2022-03-13 19:35 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

There is a new pull request by julianrother against master on the mblaze repository

https://github.com/julianrother/mblaze mthread-memory-leak
https://github.com/leahneukirchen/mblaze/pull/222

mthread: reduce memory usage
mthread keeps header data of all messages in memory until it exits without ever using it. With this change it frees the header data of each message right after processing it.

With my test data of ~100000 messages (10GB) passed to mthread with `-S` this change reduces the peak memory usage from 841MB to 27MB.

A patch file from https://github.com/leahneukirchen/mblaze/pull/222.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-mthread-memory-leak-222.patch --]
[-- Type: text/x-diff, Size: 1686 bytes --]

From 38acffc50baf70a2eb233e55c0ccc69946c38c44 Mon Sep 17 00:00:00 2001
From: Julian Rother <julian@jrother.eu>
Date: Sun, 13 Mar 2022 20:28:19 +0100
Subject: [PATCH] mthread: reduce memory usage

mthread keeps header data of all messages in memory until it exits without
ever using it. With this change it frees the header data of each message
right after processing it.
---
 mthread.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/mthread.c b/mthread.c
index 9d718fc..d200879 100644
--- a/mthread.c
+++ b/mthread.c
@@ -29,7 +29,6 @@ static int optional;
 struct container {
 	char *mid;
 	char *file;
-	struct message *msg;
 	time_t date;
 	struct container *parent;
 	struct container *child;
@@ -87,7 +86,6 @@ midcont(char *mid)
 			exit(111);
 		c->mid = mid;
 		c->file = 0;
-		c->msg = 0;
 		c->date = -1;
 		c->optional = 0;
 		c->parent = c->child = c->next = 0;
@@ -104,7 +102,6 @@ store_id(char *file, struct message *msg)
 
 	c = midcont(mid(msg));
 	c->file = strdup(file);
-	c->msg = msg;
 	c->optional = optional;
 
 	return c;
@@ -239,6 +236,8 @@ thread(char *file)
 			c->child = 0;
 		}
 	}
+
+	blaze822_free(msg);
 }
 
 time_t
@@ -290,7 +289,6 @@ find_roots()
 	top = malloc(sizeof (struct container));
 	if (!top)
 		exit(111);
-	top->msg = 0;
 	top->date = -1;
 	top->file = 0;
 	top->next = top->child = top->parent = 0;
@@ -315,7 +313,6 @@ prune_tree(struct container *c, int depth)
 			// turn into child if we don't exist and only have a child
 			c->mid = c->child->mid;
 			c->file = c->child->file;
-			c->msg = c->child->msg;
 			if (c->child->date > c->date)
 				c->date = c->child->date;
 			c->optional = c->child->optional;

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PR PATCH] [Closed]: mthread: reduce memory usage
  2022-03-13 19:35 [PR PATCH] mthread: reduce memory usage julianrother
@ 2022-03-13 19:42 ` leahneukirchen
  2022-03-13 19:42 ` leahneukirchen
  1 sibling, 0 replies; 3+ messages in thread
From: leahneukirchen @ 2022-03-13 19:42 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

There's a closed pull request on the mblaze repository

mthread: reduce memory usage
https://github.com/leahneukirchen/mblaze/pull/222

Description:
mthread keeps header data of all messages in memory until it exits without ever using it. With this change it frees the header data of each message right after processing it.

With my test data of ~100000 messages (10GB) passed to mthread with `-S` this change reduces the peak memory usage from 841MB to 27MB.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: mthread: reduce memory usage
  2022-03-13 19:35 [PR PATCH] mthread: reduce memory usage julianrother
  2022-03-13 19:42 ` [PR PATCH] [Closed]: " leahneukirchen
@ 2022-03-13 19:42 ` leahneukirchen
  1 sibling, 0 replies; 3+ messages in thread
From: leahneukirchen @ 2022-03-13 19:42 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 229 bytes --]

New comment by leahneukirchen on mblaze repository

https://github.com/leahneukirchen/mblaze/pull/222#issuecomment-1066168822

Comment:
Very good catch, this is a left-over from the previous codebase I extracted mthread from. :)

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2022-03-13 19:42 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-13 19:35 [PR PATCH] mthread: reduce memory usage julianrother
2022-03-13 19:42 ` [PR PATCH] [Closed]: " leahneukirchen
2022-03-13 19:42 ` leahneukirchen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).