[9fans] Problem with mk

[9fans] Problem with mk

[Blake McBride]

[-- Attachment #1: Type: text/plain, Size: 2686 bytes --]

Greetings,

I have spent hours on a trivial problem that I still haven't figured out. I
am using 9front, and I am building a C app using mk.  I understand about
that parallel stuff so I am calling mk with -s for now.  There are actually
three problems, but the fact that it won't build is clearly the first.
 Here are the facts:

Problem 1:

If I execute "mk -s", mk prints out what it is executing interspersed with
messages from the command being executed.  Somewhere in the middle I see:

mk: ../../bin/dpp -g -t ... : status=rc 1357: dpp 1359: Exit code 1

Each time I run it, the two numbers (1357 & 1369) are different.  This
message is showing up in the middle of all of the commands getting executed.

If I execute "mk -n" and manually execute each command the build succeeds.

If I manually execute each command and append "; echo $status" to each
command, no error code is ever displayed.

Since I can build manually, and I get no error codes when I do so, I cannot
figure out what the problem is.

Problem 2:

Even though I am executing mk with the "-s" option, it still seems like it
is running in parallel because a subsequent command can't find a file
created by a prior command - as if it didn't wait for the prior command to
finish.  Remember this build fine, and without error codes, when executed
manually.

Problem 3:

Somehow Unix or GNU "make" doesn't mix up buffered stdout with unbuffered
stderr output.  They remain in order so the total out of make and all of
the commands are shown in order and in context.  You know, so a human can
understand it.  mk appears not to handle this the same.  stdout output and
stderr output are totally out of sequence making it very, very difficult to
understand what is going on.

----

Here is my mkfile:

</$objtype/mkfile

DYNACE_PATH = ../..

BINDIR = $DYNACE_PATH/bin
LIBDIR = $DYNACE_PATH/lib
INCDIR = $DYNACE_PATH/include

TARGET = main

CLASSES = class1.d

CFILES = main.c

OBJS = ${CFILES:%.c=%.$O} ${CLASSES:%.d=%.$O}

CFLAGS = -DPLAN9 -I$INCDIR -p

C_CLASSES = ${CLASSES:%.d=%.c}

%.$O : %.c
$CC $CFLAGS $stem.c

%.c : %.d
$BINDIR/dpp -g -p $prereq

$TARGET : generics.1 generics.h generics.$O $OBJS
$LD -o $target $OBJS generics.$O $LIBDIR/Dynace.a

generics.h : $CLASSES
$BINDIR/dpp -g -t generics.h generics.c generics.$O -h -s $newprereq

generics.c : generics.h
$BINDIR/dpp -g -c

generics.1 newgens: $INCDIR/generics.h
$BINDIR/dpp -g $INCDIR/generics.h -s $CLASSES -h
touch generics.1

clean realclean:VQ:
rm -f generics.* $TARGET *.$O $C_CLASSES


Any suggestion would sure be appreciated.

Thanks.

Blake McBride

[-- Attachment #2: Type: text/html, Size: 9422 bytes --]

Re: [9fans] Problem with mk

[dexen deVries]

On Wednesday 18 of December 2013 09:23:19 Blake McBride wrote:
> 
> Problem 1:
> 


it seems you have an un-stated dependency/cies among your intermediate targets 
/ prerequisites.

Say, foo.o depends on foo.c and foo.h -- but foo.h also depends on 
generated_foo.h, which should be generated by make.

in such case, explicitly state (no recipe is necessary):


foo.h: generated_foo.h


so mk knows the `generated_foo.h' must be completed first.


> Problem 2:
> 
> Even though I am executing mk with the "-s" option, it still seems like it
> is running in parallel because a subsequent command can't find a file
> created by a prior command - as if it didn't wait for the prior command to
> finish.  Remember this build fine, and without error codes, when executed
> manually.



-s won't help you there, because it regards processing of /command line/ 
arguments, not of prerequisites. consider:

$ NPROC=1 mk my_target

also investigate -d[egp] debug stuff.


have fun with mk, it's a great little tool :-)



-- 
dexen deVries

[[[↓][→]]]

Take care of the luxuries and the necessities will take care of themselves.
                -- L. Long

Re: [9fans] Problem with mk

[Blake McBride]

[-- Attachment #1: Type: text/plain, Size: 2900 bytes --]

Thanks for the help!  See below:


On Wed, Dec 18, 2013 at 9:37 AM, dexen deVries <dexen.devries@gmail.com>wrote:

> On Wednesday 18 of December 2013 09:23:19 Blake McBride wrote:
> >
> > Problem 1:
> >
>
>
> it seems you have an un-stated dependency/cies among your intermediate
> targets
> / prerequisites.
>
> Say, foo.o depends on foo.c and foo.h -- but foo.h also depends on
> generated_foo.h, which should be generated by make.
>
> in such case, explicitly state (no recipe is necessary):
>
>
> foo.h: generated_foo.h
>
>
> so mk knows the `generated_foo.h' must be completed first.
>



Yea, actually my build process is a bit more complex than that.  What I am
doing reflects that complexity and works perfectly with make.

_Initially_ (the first time), generics.h gets built through the generics.1
target, and then updated through the generics.h target process.
 Thereafter, as the program evolves, new classes are added (*.d files) and
changed, generics.h gets updated through the generics.h process only.  The
generics.h target process reads in generics.h and updates it based on the
changed *.d files.  You can think of generics.h as a database of generic
functions that needs to be initialized and then updated as the system
evolves.

The purpose of using rules rather than explicit recipes is because the
mkfile is often used as a template for much larger projects.  The pattern
is convenient.  Larger projects just have to update the three variables
TARGET, CLASSES, and CFILES.  If I don't use recipes, the mkfile gets
unnecessarily lengthy with larger projects.

Also, in spite of my use of a recipe rather than an explicit rule, why
would that cause mk to barf?




>
>
> > Problem 2:
> >
> > Even though I am executing mk with the "-s" option, it still seems like
> it
> > is running in parallel because a subsequent command can't find a file
> > created by a prior command - as if it didn't wait for the prior command
> to
> > finish.  Remember this build fine, and without error codes, when
> executed
> > manually.
>
>
>
> -s won't help you there, because it regards processing of /command line/
> arguments, not of prerequisites. consider:
>
> $ NPROC=1 mk my_target
>
> also investigate -d[egp] debug stuff.
>

Yes, I didn't see a difference with or without the -s.  I just used it to
eliminate one possible source of problems.  Also, the random intermixing of
stdout and stderr causes me not to know what the order of execution is.


>
>
> have fun with mk, it's a great little tool :-)
>

I'm sure it is.  If I can get it to treat generics.h as a database as I
described above, it would be great.

Thanks!


>
>
>
> --
> dexen deVries
>
> [[[↓][→]]]
>
> Take care of the luxuries and the necessities will take care of themselves.
>                 -- L. Long
>
>
>

[-- Attachment #2: Type: text/html, Size: 8601 bytes --]

Re: [9fans] Problem with mk

[Blake McBride]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

The problem is solved, but I do have questions that I couldn't the answer
to in any of the docs.


On Wed, Dec 18, 2013 at 9:37 AM, dexen deVries <dexen.devries@gmail.com>wrote:
>
>
> -s won't help you there, because it regards processing of /command line/
> arguments, not of prerequisites. consider:
>
> $ NPROC=1 mk my_target
>


The NPROC=1 solved the problem.  My mkfile is written in a way that
requires a sequential operation.  I need to look at it further.  Think a
large part of my problem was a misunderstanding of the -s option.  Anyway,
here is a question.  Presuming I can't fix my mkfile to operate in
parallel, is there a way to cause sequential operation other than the
"NPROC=1" at the command line?  It's too easy to forget.  Is there a way to
specify that mode of operation by something I put into the mkfile?

Thanks!

Blake

[-- Attachment #2: Type: text/html, Size: 1742 bytes --]

Re: [9fans] Problem with mk

[Jacob Todd]

[-- Attachment #1: Type: text/plain, Size: 38 bytes --]

You could put NPROC=1 in the mkfile.

[-- Attachment #2: Type: text/html, Size: 55 bytes --]

Re: [9fans] Problem with mk

[Blake McBride]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

Thanks.  That is what I was wondering.  In the interim, and having a better
understanding of what is going on, I was able to re-work my mkfile to
support parallel builds where possible.  I am, however, experiencing a new
problem.  I will write that up in a separate post.

Thank for the info!

Blake



On Wed, Dec 18, 2013 at 3:04 PM, Jacob Todd <jaketodd422@gmail.com> wrote:

> You could put NPROC=1 in the mkfile.
>

[-- Attachment #2: Type: text/html, Size: 799 bytes --]

Re: [9fans] Problem with mk

[dexen deVries]

the problem is not solved, merely one of the manifestation is eliminated.
actual solution requires stating all the dependencies.

if your list of sources and/or targets is dynamic, consider using mk include:
<| GEN_DEPS
where `GEN_DEPS' is your script generating deps in form TARGET: PREREQUISITE

On Wed, Dec 18, 2013 at 10:14 PM, Blake McBride <blake@mcbride.name> wrote:
> Thanks.  That is what I was wondering.  In the interim, and having a better
> understanding of what is going on, I was able to re-work my mkfile to
> support parallel builds where possible.  I am, however, experiencing a new
> problem.  I will write that up in a separate post.
>
> Thank for the info!
>
> Blake
>
>
>
> On Wed, Dec 18, 2013 at 3:04 PM, Jacob Todd <jaketodd422@gmail.com> wrote:
>>
>> You could put NPROC=1 in the mkfile.
>
>

Re: [9fans] Problem with mk

[Blake McBride]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

On Thu, Dec 19, 2013 at 2:16 AM, dexen deVries <dexen.devries@gmail.com>wrote:

> the problem is not solved, merely one of the manifestation is eliminated.
> actual solution requires stating all the dependencies.
>
> if your list of sources and/or targets is dynamic, consider using mkinclude:
> <| GEN_DEPS
> where `GEN_DEPS' is your script generating deps in form TARGET:
> PREREQUISITE
>
>
The following mkfile does what I need/expect without explicit dependencies.

# Makefile for Plan 9

</$objtype/mkfile

DYNACE_PATH = ../..

BINDIR = $DYNACE_PATH/bin
LIBDIR = $DYNACE_PATH/lib
INCDIR = $DYNACE_PATH/include

TARGET = main

CLASSES = class1.d

CFILES = main.c

OBJS = ${CFILES:%.c=%.$O} ${CLASSES:%.d=%.$O}

CFLAGS = -DPLAN9 -I$INCDIR -p

C_CLASSES = ${CLASSES:%.d=%.c}

%.$O : %.c
$CC $CFLAGS $stem.c

%.c : %.d
$BINDIR/dpp -g -p $stem.d

$TARGET : generics.$O $OBJS
$LD -o $target $OBJS generics.$O $LIBDIR/Dynace.a

generics.h : generics.1 $CLASSES
$BINDIR/dpp -g -t generics.h generics.c generics.$O -h -s $newprereq

generics.c : generics.h
$BINDIR/dpp -g -c

generics.1 newgens: $INCDIR/generics.h
$BINDIR/dpp -g $INCDIR/generics.h -s $CLASSES -h
touch generics.1

$OBJS $C_CLASSES : generics.h

clean realclean:VQ:
rm -f generics.* $TARGET *.$O $C_CLASSES

[-- Attachment #2: Type: text/html, Size: 6976 bytes --]

Re: [9fans] Problem with mk

[Blake McBride]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

On Wed, Dec 18, 2013 at 9:23 AM, Blake McBride <blake@mcbride.name> wrote:
>
>
> Problem 3:
>
> Somehow Unix or GNU "make" doesn't mix up buffered stdout with unbuffered
> stderr output.  They remain in order so the total out of make and all of
> the commands are shown in order and in context.  You know, so a human can
> understand it.  mk appears not to handle this the same.  stdout output
> and stderr output are totally out of sequence making it very, very
> difficult to understand what is going on.
>
>
mk should flush stdout after completion each command.

[-- Attachment #2: Type: text/html, Size: 1121 bytes --]

Re: [9fans] Problem with mk

[Bakul Shah]

On Wed, 18 Dec 2013 10:11:22 CST Blake McBride <blake@mcbride.name> wrote:
> >
> > Somehow Unix or GNU "make" doesn't mix up buffered stdout with unbuffered
> > stderr output.  They remain in order so the total out of make and all of
> > the commands are shown in order and in context.  You know, so a human can
> > understand it.  mk appears not to handle this the same.  stdout output
> > and stderr output are totally out of sequence making it very, very
> > difficult to understand what is going on.
> >
> >
> mk should flush stdout after completion each command.

Just as plan9 is not unix, mk is not make. Rather than expect
it to behave like make, it pays to understand how mk works,
may be by experimenting with little mkfiles.

Re: [9fans] Problem with mk

[Aram Hăvărneanu]

D on Plan 9? This is madness.

-- 
Aram Hăvărneanu

Re: [9fans] Problem with mk

[Blake McBride]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

This is not D.  It is a language of my own design called Dynace.  It adds
multiple inheritance, a metaobject protocol, generic functions, weak
typing, garbage collection all to standard C.  Class users (*.c files) are
standard C in every way, there is only one data type added.  The system is
written entirely in portable C, and has been used in production
environments for years.  It also comes with a 350 page book that completely
documents it, and a tutorial.  I've got it running on Plan-9 except for
this mkfile issue with the examples.

The system is open-source.  A new release will come out with Plan-9 support
as soon as I get the mkfile issue solved.

Thanks.






On Wed, Dec 18, 2013 at 11:06 AM, Aram Hăvărneanu <aram.h@mgk.ro> wrote:

> D on Plan 9? This is madness.
>
> --
> Aram Hăvărneanu
>
>

[-- Attachment #2: Type: text/html, Size: 1469 bytes --]

Re: [9fans] Problem with mk

[Rubén Berenguel]

[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]

Seeing a Go link on your homepage I thought, oh, you also program in go?
And then found that you actually *play* go (Google harmed a lot looking for
information on the game). More surprising :D


On Wed, Dec 18, 2013 at 6:18 PM, Blake McBride <blake@mcbride.name> wrote:

> This is not D.  It is a language of my own design called Dynace.  It adds
> multiple inheritance, a metaobject protocol, generic functions, weak
> typing, garbage collection all to standard C.  Class users (*.c files)
> are standard C in every way, there is only one data type added.  The system
> is written entirely in portable C, and has been used in production
> environments for years.  It also comes with a 350 page book that completely
> documents it, and a tutorial.  I've got it running on Plan-9 except for
> this mkfile issue with the examples.
>
> The system is open-source.  A new release will come out with Plan-9
> support as soon as I get the mkfile issue solved.
>
> Thanks.
>
>
>
>
>
>
> On Wed, Dec 18, 2013 at 11:06 AM, Aram Hăvărneanu <aram.h@mgk.ro> wrote:
>
>> D on Plan 9? This is madness.
>>
>> --
>> Aram Hăvărneanu
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 1948 bytes --]

Re: [9fans] Problem with mk

[Blake McBride]

[-- Attachment #1: Type: text/plain, Size: 1765 bytes --]

Often there is a set of reasonable, alternative algorithms one may utilize
to solve a problem.  Then, there is a set of unreasonable algorithms one
may employ that may or may not solve the problem given various conditions.
 "make" has a proven reasonable algorithm to solve the problem for which it
was designed.  It employs one algorithm among other possible, and equally
good, algorithms.  You are implying that mk is using another valid
algorithm rather than an unreasonable one.  I can't see it yet.

I'd be a better judge if I understood the purposeful, thought out reason
behind the problems I am experiencing - assuming there is one.  "That's
just the way it works" or "we do it differently because we are not unix"
are stupid as hell arguments.

I remain confident that there is a thought out, reasonable algorithm
employed by mk that I am yet ignorant of.

Blake


On Wed, Dec 18, 2013 at 10:40 AM, Bakul Shah <bakul@bitblocks.com> wrote:

> On Wed, 18 Dec 2013 10:11:22 CST Blake McBride <blake@mcbride.name> wrote:
> > >
> > > Somehow Unix or GNU "make" doesn't mix up buffered stdout with
> unbuffered
> > > stderr output.  They remain in order so the total out of make and all
> of
> > > the commands are shown in order and in context.  You know, so a human
> can
> > > understand it.  mk appears not to handle this the same.  stdout output
> > > and stderr output are totally out of sequence making it very, very
> > > difficult to understand what is going on.
> > >
> > >
> > mk should flush stdout after completion each command.
>
> Just as plan9 is not unix, mk is not make. Rather than expect
> it to behave like make, it pays to understand how mk works,
> may be by experimenting with little mkfiles.
>
>

[-- Attachment #2: Type: text/html, Size: 4018 bytes --]

Re: [9fans] Problem with mk

[Kurt H Maier]

Quoting Blake McBride <blake@mcbride.name>:

> I'd be a better judge if I understood the purposeful, thought out reason
> behind the problems I am experiencing - assuming there is one.  "That's
> just the way it works" or "we do it differently because we are not unix"
> are stupid as hell arguments.

No they aren't.

Meanwhile, I trust you have read "Maintaining Files on Plan 9 with Mk" [1]
and "Plan 9 Mkfiles" [2] in their entirety.  Whining about software not
meeting your behavioral expectations will get you nowhere.  Just because
other build systems make it relatively easy for you to shove unintended
functionality into tools is no reason to disparage software that is working
as intended.

khm


[1] http://doc.cat-v.org/plan_9/4th_edition/papers/mk
[2] http://doc.cat-v.org/plan_9/4th_edition/papers/mkfiles

Re: [9fans] Problem with mk

[Bakul Shah]

On Wed, 18 Dec 2013 11:28:21 CST Blake McBride <blake@mcbride.name> wrote:
>
> I'd be a better judge if I understood the purposeful, thought out reason
> behind the problems I am experiencing - assuming there is one.  "That's
> just the way it works" or "we do it differently because we are not unix"
> are stupid as hell arguments.

"That's just the way it works" is exactly the right attitude
to take when you are learning a new language or entering a new
culture! Expecting the new language/culture/world to be better
or similar to the old will lead to a lot of frustration and
slow you down quite a bit. So leave your value judgements at
the door and just try to understand things *as they are*.

> I remain confident that there is a thought out, reasonable algorithm
> employed by mk that I am yet ignorant of.

		Tom Knight and the Lisp Machine

    A novice was trying to fix a broken Lisp machine by
    turning the power off and on.

    Knight, seeing what the student was doing, spoke sternly:
    You cannot fix a machine by just power-cycling it with no
    understanding of what is going wrong.

    Knight turned the machine off and on.

    The machine worked.

Re: [9fans] Problem with mk

[Blake McBride]

[-- Attachment #1: Type: text/plain, Size: 1686 bytes --]

In response to both of you:

On Wed, Dec 18, 2013 at 11:28 AM, Blake McBride <blake@mcbride.name> wrote:

> ...
>
> I remain confident that there is a thought out, reasonable algorithm
> employed by mk that I am yet ignorant of.
>
> Blake
>
>
If you could point me in the right direction, I'd surely appreciate it.


On Wed, Dec 18, 2013 at 12:13 PM, Bakul Shah <bakul@bitblocks.com> wrote:

> On Wed, 18 Dec 2013 11:28:21 CST Blake McBride <blake@mcbride.name> wrote:
> >
> > I'd be a better judge if I understood the purposeful, thought out reason
> > behind the problems I am experiencing - assuming there is one.  "That's
> > just the way it works" or "we do it differently because we are not unix"
> > are stupid as hell arguments.
>
> "That's just the way it works" is exactly the right attitude
> to take when you are learning a new language or entering a new
> culture! Expecting the new language/culture/world to be better
> or similar to the old will lead to a lot of frustration and
> slow you down quite a bit. So leave your value judgements at
> the door and just try to understand things *as they are*.
>
> > I remain confident that there is a thought out, reasonable algorithm
> > employed by mk that I am yet ignorant of.
>
>                 Tom Knight and the Lisp Machine
>
>     A novice was trying to fix a broken Lisp machine by
>     turning the power off and on.
>
>     Knight, seeing what the student was doing, spoke sternly:
>     You cannot fix a machine by just power-cycling it with no
>     understanding of what is going wrong.
>
>     Knight turned the machine off and on.
>
>     The machine worked.
>
>

[-- Attachment #2: Type: text/html, Size: 4000 bytes --]

Re: [9fans] Problem with mk

[Blake McBride]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

In particular, why would manual execution while echoing $status (to be sure
no errors are reported) work, but mk execution of the exact same sequence not
work?

Thanks!

Blake



On Wed, Dec 18, 2013 at 12:20 PM, Blake McBride <blake@mcbride.name> wrote:

> In response to both of you:
>
> On Wed, Dec 18, 2013 at 11:28 AM, Blake McBride <blake@mcbride.name>
>  wrote:
>
>> ...
>>
>>
>> I remain confident that there is a thought out, reasonable algorithm
>> employed by mk that I am yet ignorant of.
>>
>> Blake
>>
>>
> If you could point me in the right direction, I'd surely appreciate it.
>
>

[-- Attachment #2: Type: text/html, Size: 1915 bytes --]