[Caml-list] Common code over different fields of a record, using macros?

[Caml-list] Common code over different fields of a record, using macros?

[Anthony Tavener]

[-- Attachment #1: Type: text/plain, Size: 2484 bytes --]

I find that records often result in redundant code... the same operation
being specified for different fields (of the same type). I could use arrays
in these cases, but then lose meaningful (to programmers) field names.

I tried using camlp4 (and camlp5) macros to get the effect of passing a
field "name" to a "function". I could get my example to work if I update
mutable fields (record.field <- x), but using the functional record-update
{record with field=x} doesn't work...


--- Simplified mock-up of my current situation (not enough to compile) ---

(* a record with some fields of the same type... imagine there might be many
more fields *)
type stats = { strength: int; agility: int }

(* macro which doesn't work... "field" which follows "with" doesn't get
replaced *)
DEFINE AGE(field) = fun (state,age) ->
  let s,a = age_stat state.field age.field in
  {state with field=s}, {age with field=a}

(* val f : (stats * stats) -> int -> (stats * stats) *)
let age_character input = function
  | n when n < 2 -> input
  | 2 -> AGE(strength) input
  | 3 -> AGE(agility) input
  | _ -> AGE2(strength,agility) input

(* a mock-up of usage... *)
let state = { strength=3; agility=1 } in
let age = { strength=0; agility=0 } in
let state',age' = age_character (state,age) (rand 8) in ...

---

After processing by camlp4 with macros this is what the first case becomes:

  | 1 -> let s,a = age_stat state.strength age.strength in
         {state with field=s}, {age with field=a}

"field" isn't replaced with "strength" in the record-update. I tried looking
at Camlp4MacroParser.ml... but it makes my head swim. It must be doing
something much more careful than literal text replacement, but perhaps too
careful... or incomplete? Does anyone know how these macros work? Is this
proper behavior for some reason, or an unhandled edge case?

The problem is I don't want a block of code like this to be repeated 10
times with only a field-name change for each (and 4 field names each time!).
That's not well readable, prone to error, and harder to maintain properly.

Sometimes I wish I could present an alternative view of the same data, such
as having an array "view" into part of a record... verified to be typesafe
by the compiler... and compiled into the same simple offsets in the end.
Maybe it's my asm/C origins which I never seem to escape. I mention this in
hope that someone says "Oh, that's exactly what you can do... like this!" :)

Thank-you for any help caml-list!

 Tony

[-- Attachment #2: Type: text/html, Size: 2787 bytes --]

Re: [Caml-list] Common code over different fields of a record, using macros?

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 8123 bytes --]

1. This discontinuity (field names in access position get replaced, but not
in a "with ...") is a bug of the current macro extension of Camlp4. I have
just written a patch that fixes this precise issue.
The good news is that you don't need to patch and recompile the whole OCaml
distribution to benefit from the fix. The "macro parser" (the part of camlp4
that supports DEFINE and so on) is a camlp4 extension like any other, so you
just have to compile the modified code somewhere and pass the ".cmo" to the
camlp4* preprocessor you run on your source code.
The bad news is that while the proposed patch fixes your particular use
case, I have found other similar inconsistencies -- eg. use in pattern
position -- that are harder to fix and would require a deeper
re-implementation of this syntax extension logic. I know how I should do
that but, unless someone is specifically interested in fixing this, I'm not
sure I will actually do it.

My patch is distributed as an individual source file, which is a modified
version of the original Camlp4Parsers/Camlp4MacroParser.ml file of the ocaml
distribution :
  http://bluestorm.info/camlp4/dev/define_and_fields_bug/macro.ml.html
I use a small test file exhibiting your use case, an other use fixed by the
change, an unrelated bug, and an issue which isn't fixed by the change.
  http://bluestorm.info/camlp4/dev/define_and_fields_bug/test.ml.html
All source files are available directly
  http://bluestorm.info/camlp4/dev/define_and_fields_bug/list.php

It's easy to compare test.ml against both the original and the modified
macro parser. After compilation of macro.ml (ocamlfind-using compilation
command given at the top of the file), use :
  `camlp4of test.ml` for processing with the original macro parser
  `camlp4o macro.cmo` for processing with the modifier macro parser

Hope this helps.

PS : I will eventually file a ticket on the ocaml bug tracker on this
Camlp4MacroParser issue. Xavier Clerc is quite responsive on these kind of
things, and if I (or someone else; don't hesitate!) come with a reasonable
fix it may get into 3.13.


2. As a more general answer to your post, you should keep in mind that the
macro facility, as implemented by Camlp4Parsers/Camlp4MacroParser.ml , is
not intended to be an all-powerful syntax extension facility for OCaml. It
is rather a simple example of the power of writing Camlp4 extensions, that
provides simple substitute-and-replace facilities for expressions in
patterns. The particular misbehavior you encountered should be considered a
bug and it is legitimate to try to fix it, but it doesn't mean that this
macro definition facility ever will suit all your potential needs of clever
metaprogramming. Don't expect a much finer support for syntactic classes ("i
want this parameter to accept all syntactically valid patterns, and nothing
more") or hygienic macros any time soon.
If you want a powerful syntactical metaprogramming tool, your best bet would
be to use Camlp4 directly. While still limited, it is much more powerful
that the simple Camlp4MacroParser facility. I'm not sure however that
relying heavily on such syntactic facilities is a good idea in the long
turn; at least, you shouldn't expect other developers to easily accept to
use your non-standard extensions. However, for projects that you develop
alone, or where you have strong control on the compilation chain, camlp4 (or
camlp5) can be a nice tool.


3. In your specific example, a good compromise would be to use a hashtable
indexed by an algebraic datatype.

  type field = Strength | Agility

  type stats_table = (field, int) Hashtbl.t

  let age_state _ _ = failwith "not implemented"

  let _AGE field = fun (state, age) ->
    let s,a = age_state (Hashtbl.find state field) (Hashtbl.find age field)
in
    Hashtbl.add state field s;
    Hashtbl.add age field a;
    (state, age)

  let age_character input = function
    | n when n < 2 -> input
    | 2 -> _AGE(Strength) input
    | _ -> _AGE(Agility) input

In my example, an impure data structure is used (Hashtbl) and updated
imperatively; if you want to keep a pure update-by-copy operation like the
{... with ..} facility of records, you should rather use Map instead of
Hashtbl. As thread all state through a state monad, I considered this was
not an issue.

If you want a more concise syntax for hasthable access (and do not use the
related syntax extension), you can define infix operators, following an idea
from Paolo Donadeo:

  let (-->) table key = Hashtbl.find table key
  let (<--) table (key, value) = Hashtbl.add table key value

The downside of associative data structure instead of records here is that
force each field to have the same type. You couldn't have strength be an int
and agility be a float. It is fine however in the common situation where you
can have a uniform operation over field names because they are all used with
the same type.

Finally, it can be a good idea to determine a confined part of your program
where you need that indexing flexibility, where you would use that
"stats_table" representation, and convert in and out to a record "stats"
type so that the rest of your program see the more standard record type.
In my experience, the less typed but more flexible representations are often
only useful in the "ends" of your application, typically the place where you
do input-output; a common case is when constructing the data by parsing a
configuration file.


On Thu, Feb 17, 2011 at 12:53 AM, Anthony Tavener <anthony.tavener@gmail.com
> wrote:

> I find that records often result in redundant code... the same operation
> being specified for different fields (of the same type). I could use arrays
> in these cases, but then lose meaningful (to programmers) field names.
>
> I tried using camlp4 (and camlp5) macros to get the effect of passing a
> field "name" to a "function". I could get my example to work if I update
> mutable fields (record.field <- x), but using the functional record-update
> {record with field=x} doesn't work...
>
>
> --- Simplified mock-up of my current situation (not enough to compile) ---
>
> (* a record with some fields of the same type... imagine there might be
> many more fields *)
> type stats = { strength: int; agility: int }
>
> (* macro which doesn't work... "field" which follows "with" doesn't get
> replaced *)
> DEFINE AGE(field) = fun (state,age) ->
>   let s,a = age_stat state.field age.field in
>   {state with field=s}, {age with field=a}
>
> (* val f : (stats * stats) -> int -> (stats * stats) *)
> let age_character input = function
>   | n when n < 2 -> input
>   | 2 -> AGE(strength) input
>   | 3 -> AGE(agility) input
>   | _ -> AGE2(strength,agility) input
>
> (* a mock-up of usage... *)
> let state = { strength=3; agility=1 } in
> let age = { strength=0; agility=0 } in
> let state',age' = age_character (state,age) (rand 8) in ...
>
> ---
>
> After processing by camlp4 with macros this is what the first case becomes:
>
>   | 1 -> let s,a = age_stat state.strength age.strength in
>          {state with field=s}, {age with field=a}
>
> "field" isn't replaced with "strength" in the record-update. I tried
> looking at Camlp4MacroParser.ml... but it makes my head swim. It must be
> doing something much more careful than literal text replacement, but perhaps
> too careful... or incomplete? Does anyone know how these macros work? Is
> this proper behavior for some reason, or an unhandled edge case?
>
> The problem is I don't want a block of code like this to be repeated 10
> times with only a field-name change for each (and 4 field names each time!).
> That's not well readable, prone to error, and harder to maintain properly.
>
> Sometimes I wish I could present an alternative view of the same data, such
> as having an array "view" into part of a record... verified to be typesafe
> by the compiler... and compiled into the same simple offsets in the end.
> Maybe it's my asm/C origins which I never seem to escape. I mention this in
> hope that someone says "Oh, that's exactly what you can do... like this!" :)
>
> Thank-you for any help caml-list!
>
>  Tony
>
>

[-- Attachment #2: Type: text/html, Size: 9295 bytes --]

Re: [Caml-list] Common code over different fields of a record, using macros?

[Anthony Tavener]

That explains everything, thank-you Gabriel for providing a patch. I
will be modest in my expectations from the macro parser. It does help
to understand that it's more of an example or proof-of-concept.

I've been looking more seriously into camlp4 now and will continue.
Also, your cautionary points about too much syntactic alteration are
well noted. I'd rather keep things as stock as possible. Though I
always hope for a performant and expressive language which provides
good (compile-time) metaprogramming facilities. OCaml has 2.5 out of
3. ;) Enough that I'm happy. :)

A note on the hashtable approach... that is how most of my data is
stored, though in larger blocks. One table is these records of
characteristics. There's a certain ideal in being able to access a
variable by key, but I'm worried about getting the granularity too
fine and having problems with time/space costs in the long run. Your
suggestion is tempting though. I also am trying to keep more purity in
low-level functions, while the outer levels handle gathering from or
updating the database (hashtables).

My current solution, after a reply from G.Yziquel, is a set of
accessor functions which makes the high-level use rather succinct and
clear, but rather verbose in support functions...

(* in Stat module... this provides means for a read-compute-update *)
module Stat = struct
  let strength s = (s.strength,fun x -> {s with strength=x})
  (* and more... *)
end

(* this part is a little clunky due to all my tuple returns *)
let age access (state,age) =
  let s,sets = access state  (* 'access' gives us the current value
and a way to create an updated record *)
  and a,seta = access age in
  let s,a = age_stat s a in
  sets s,seta a

(* usage of 'age' with accessor function *)
  age Stat.strength input

A little messy, but this doesn't need any syntax change and redundancy
factored out. I will continue learning camlp4 for possible suitability
in this and other situations (keeping in mind that acceptable 'plain
OCaml' solutions would be preferable!).

And you make another great point about the 'ends' of an application
often dealing with 'untyped' or raw data. I might need conversions
anyway, and could possibly leverage them in other places if
appropriate.

Thank-you again for your very in-depth reply!


On Fri, Feb 18, 2011 at 2:11 AM, Gabriel Scherer
<gabriel.scherer@gmail.com> wrote:
> 1. This discontinuity (field names in access position get replaced, but not
> in a "with ...") is a bug of the current macro extension of Camlp4. I have
> just written a patch that fixes this precise issue.
> The good news is that you don't need to patch and recompile the whole OCaml
> distribution to benefit from the fix. The "macro parser" (the part of camlp4
> that supports DEFINE and so on) is a camlp4 extension like any other, so you
> just have to compile the modified code somewhere and pass the ".cmo" to the
> camlp4* preprocessor you run on your source code.
> The bad news is that while the proposed patch fixes your particular use
> case, I have found other similar inconsistencies -- eg. use in pattern
> position -- that are harder to fix and would require a deeper
> re-implementation of this syntax extension logic. I know how I should do
> that but, unless someone is specifically interested in fixing this, I'm not
> sure I will actually do it.
>
> My patch is distributed as an individual source file, which is a modified
> version of the original Camlp4Parsers/Camlp4MacroParser.ml file of the ocaml
> distribution :
>   http://bluestorm.info/camlp4/dev/define_and_fields_bug/macro.ml.html
> I use a small test file exhibiting your use case, an other use fixed by the
> change, an unrelated bug, and an issue which isn't fixed by the change.
>   http://bluestorm.info/camlp4/dev/define_and_fields_bug/test.ml.html
> All source files are available directly
>   http://bluestorm.info/camlp4/dev/define_and_fields_bug/list.php
>
> It's easy to compare test.ml against both the original and the modified
> macro parser. After compilation of macro.ml (ocamlfind-using compilation
> command given at the top of the file), use :
>   `camlp4of test.ml` for processing with the original macro parser
>   `camlp4o macro.cmo` for processing with the modifier macro parser
>
> Hope this helps.
>
> PS : I will eventually file a ticket on the ocaml bug tracker on this
> Camlp4MacroParser issue. Xavier Clerc is quite responsive on these kind of
> things, and if I (or someone else; don't hesitate!) come with a reasonable
> fix it may get into 3.13.
>
>
> 2. As a more general answer to your post, you should keep in mind that the
> macro facility, as implemented by Camlp4Parsers/Camlp4MacroParser.ml , is
> not intended to be an all-powerful syntax extension facility for OCaml. It
> is rather a simple example of the power of writing Camlp4 extensions, that
> provides simple substitute-and-replace facilities for expressions in
> patterns. The particular misbehavior you encountered should be considered a
> bug and it is legitimate to try to fix it, but it doesn't mean that this
> macro definition facility ever will suit all your potential needs of clever
> metaprogramming. Don't expect a much finer support for syntactic classes ("i
> want this parameter to accept all syntactically valid patterns, and nothing
> more") or hygienic macros any time soon.
> If you want a powerful syntactical metaprogramming tool, your best bet would
> be to use Camlp4 directly. While still limited, it is much more powerful
> that the simple Camlp4MacroParser facility. I'm not sure however that
> relying heavily on such syntactic facilities is a good idea in the long
> turn; at least, you shouldn't expect other developers to easily accept to
> use your non-standard extensions. However, for projects that you develop
> alone, or where you have strong control on the compilation chain, camlp4 (or
> camlp5) can be a nice tool.
>
>
> 3. In your specific example, a good compromise would be to use a hashtable
> indexed by an algebraic datatype.
>
>   type field = Strength | Agility
>
>   type stats_table = (field, int) Hashtbl.t
>
>   let age_state _ _ = failwith "not implemented"
>
>   let _AGE field = fun (state, age) ->
>     let s,a = age_state (Hashtbl.find state field) (Hashtbl.find age field)
> in
>     Hashtbl.add state field s;
>     Hashtbl.add age field a;
>     (state, age)
>
>   let age_character input = function
>     | n when n < 2 -> input
>     | 2 -> _AGE(Strength) input
>     | _ -> _AGE(Agility) input
>
> In my example, an impure data structure is used (Hashtbl) and updated
> imperatively; if you want to keep a pure update-by-copy operation like the
> {... with ..} facility of records, you should rather use Map instead of
> Hashtbl. As thread all state through a state monad, I considered this was
> not an issue.
>
> If you want a more concise syntax for hasthable access (and do not use the
> related syntax extension), you can define infix operators, following an idea
> from Paolo Donadeo:
>
>   let (-->) table key = Hashtbl.find table key
>   let (<--) table (key, value) = Hashtbl.add table key value
>
> The downside of associative data structure instead of records here is that
> force each field to have the same type. You couldn't have strength be an int
> and agility be a float. It is fine however in the common situation where you
> can have a uniform operation over field names because they are all used with
> the same type.
>
> Finally, it can be a good idea to determine a confined part of your program
> where you need that indexing flexibility, where you would use that
> "stats_table" representation, and convert in and out to a record "stats"
> type so that the rest of your program see the more standard record type.
> In my experience, the less typed but more flexible representations are often
> only useful in the "ends" of your application, typically the place where you
> do input-output; a common case is when constructing the data by parsing a
> configuration file.
>
>
> On Thu, Feb 17, 2011 at 12:53 AM, Anthony Tavener
> <anthony.tavener@gmail.com> wrote:
>>
>> I find that records often result in redundant code... the same operation
>> being specified for different fields (of the same type). I could use arrays
>> in these cases, but then lose meaningful (to programmers) field names.
>>
>> I tried using camlp4 (and camlp5) macros to get the effect of passing a
>> field "name" to a "function". I could get my example to work if I update
>> mutable fields (record.field <- x), but using the functional record-update
>> {record with field=x} doesn't work...
>>
>>
>> --- Simplified mock-up of my current situation (not enough to compile) ---
>>
>> (* a record with some fields of the same type... imagine there might be
>> many more fields *)
>> type stats = { strength: int; agility: int }
>>
>> (* macro which doesn't work... "field" which follows "with" doesn't get
>> replaced *)
>> DEFINE AGE(field) = fun (state,age) ->
>>   let s,a = age_stat state.field age.field in
>>   {state with field=s}, {age with field=a}
>>
>> (* val f : (stats * stats) -> int -> (stats * stats) *)
>> let age_character input = function
>>   | n when n < 2 -> input
>>   | 2 -> AGE(strength) input
>>   | 3 -> AGE(agility) input
>>   | _ -> AGE2(strength,agility) input
>>
>> (* a mock-up of usage... *)
>> let state = { strength=3; agility=1 } in
>> let age = { strength=0; agility=0 } in
>> let state',age' = age_character (state,age) (rand 8) in ...
>>
>> ---
>>
>> After processing by camlp4 with macros this is what the first case
>> becomes:
>>
>>   | 1 -> let s,a = age_stat state.strength age.strength in
>>          {state with field=s}, {age with field=a}
>>
>> "field" isn't replaced with "strength" in the record-update. I tried
>> looking at Camlp4MacroParser.ml... but it makes my head swim. It must be
>> doing something much more careful than literal text replacement, but perhaps
>> too careful... or incomplete? Does anyone know how these macros work? Is
>> this proper behavior for some reason, or an unhandled edge case?
>>
>> The problem is I don't want a block of code like this to be repeated 10
>> times with only a field-name change for each (and 4 field names each time!).
>> That's not well readable, prone to error, and harder to maintain properly.
>>
>> Sometimes I wish I could present an alternative view of the same data,
>> such as having an array "view" into part of a record... verified to be
>> typesafe by the compiler... and compiled into the same simple offsets in the
>> end. Maybe it's my asm/C origins which I never seem to escape. I mention
>> this in hope that someone says "Oh, that's exactly what you can do... like
>> this!" :)
>>
>> Thank-you for any help caml-list!
>>
>>  Tony
>>
>
>