[PR PATCH] mthread: reduce memory usage

[PR PATCH] mthread: reduce memory usage

[julianrother]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

There is a new pull request by julianrother against master on the mblaze repository

https://github.com/julianrother/mblaze mthread-memory-leak
https://github.com/leahneukirchen/mblaze/pull/222

mthread: reduce memory usage
mthread keeps header data of all messages in memory until it exits without ever using it. With this change it frees the header data of each message right after processing it.

With my test data of ~100000 messages (10GB) passed to mthread with `-S` this change reduces the peak memory usage from 841MB to 27MB.

A patch file from https://github.com/leahneukirchen/mblaze/pull/222.patch is attached

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: github-pr-mthread-memory-leak-222.patch --]
[-- Type: text/x-diff, Size: 1686 bytes --]

From 38acffc50baf70a2eb233e55c0ccc69946c38c44 Mon Sep 17 00:00:00 2001
From: Julian Rother <julian@jrother.eu>
Date: Sun, 13 Mar 2022 20:28:19 +0100
Subject: [PATCH] mthread: reduce memory usage

mthread keeps header data of all messages in memory until it exits without
ever using it. With this change it frees the header data of each message
right after processing it.
---
 mthread.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/mthread.c b/mthread.c
index 9d718fc..d200879 100644
--- a/mthread.c
+++ b/mthread.c
@@ -29,7 +29,6 @@ static int optional;
 struct container {
 	char *mid;
 	char *file;
-	struct message *msg;
 	time_t date;
 	struct container *parent;
 	struct container *child;
@@ -87,7 +86,6 @@ midcont(char *mid)
 			exit(111);
 		c->mid = mid;
 		c->file = 0;
-		c->msg = 0;
 		c->date = -1;
 		c->optional = 0;
 		c->parent = c->child = c->next = 0;
@@ -104,7 +102,6 @@ store_id(char *file, struct message *msg)
 
 	c = midcont(mid(msg));
 	c->file = strdup(file);
-	c->msg = msg;
 	c->optional = optional;
 
 	return c;
@@ -239,6 +236,8 @@ thread(char *file)
 			c->child = 0;
 		}
 	}
+
+	blaze822_free(msg);
 }
 
 time_t
@@ -290,7 +289,6 @@ find_roots()
 	top = malloc(sizeof (struct container));
 	if (!top)
 		exit(111);
-	top->msg = 0;
 	top->date = -1;
 	top->file = 0;
 	top->next = top->child = top->parent = 0;
@@ -315,7 +313,6 @@ prune_tree(struct container *c, int depth)
 			// turn into child if we don't exist and only have a child
 			c->mid = c->child->mid;
 			c->file = c->child->file;
-			c->msg = c->child->msg;
 			if (c->child->date > c->date)
 				c->date = c->child->date;
 			c->optional = c->child->optional;

Re: [PR PATCH] [Closed]: mthread: reduce memory usage

[leahneukirchen]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

There's a closed pull request on the mblaze repository

mthread: reduce memory usage
https://github.com/leahneukirchen/mblaze/pull/222

Description:
mthread keeps header data of all messages in memory until it exits without ever using it. With this change it frees the header data of each message right after processing it.

With my test data of ~100000 messages (10GB) passed to mthread with `-S` this change reduces the peak memory usage from 841MB to 27MB.

Re: mthread: reduce memory usage

[leahneukirchen]

[-- Attachment #1: Type: text/plain, Size: 229 bytes --]

New comment by leahneukirchen on mblaze repository

https://github.com/leahneukirchen/mblaze/pull/222#issuecomment-1066168822

Comment:
Very good catch, this is a left-over from the previous codebase I extracted mthread from. :)