This is a design proposal for a general purpose feature, but I'm going to explain in terms of the FWT since it is what is driving the short term need. This is a very long post, but the conclusion is a pretty radical change proposal (I actually reached the conclusion by act of writing the post - so its a journey :-)

I'm at a point in the FWT development, where I need to solve a core problem before moving forward. Widgets are a tree data structure. Typically the leaves of the tree are widgets like labels, buttons, and text fields. In the FWT I'm calling the container widgets panes and they are responsible for layout.

There are typically two kinds of panes: those with a fixed number of children and those with an open number of children. Consider an EdgePane which has up to five children: top, bottom, left, right, or center. An the other hand, GridPane has an open number of children to be laid out in a grid - might be 2 or 200 children. Both of these are types which manage a collection of child widgets. But they aren't just simple collections, they have their own configuration fields too. What I'm currently doing is something like this:

EdgePane
{
  add(GridPane
  {
    layout = "left"
    numCols = 1
    add(Label { text = "alpha" })
    add(Label { text = "beta" })
  })
  add(GridPane
  {
    layout = "right"
    numCols = 2
    add(Label { text = "gamma" })
    add(Label { text = "delta" })
  })
}

The big problem is that I'm using procedural code via the add method. This works OK in code, but won't make for clean serialization because it is not purely declarative. To do this with a pure declarative model might look like this:

EdgePane
{
  children =
  [
    GridPane
    {
      layout = "left"
      numCols = 1
      children =
      [  
        Label { text = "alpha" })
        Label { text = "beta" })
      ]
    },
    GridPane
    {
      layout = "right"
      numCols = 2
      children =
      [  
        Label { text = "gamma" })
        Label { text = "delta" })
      ]
    }
  ]
}

But now my model is artificially twice as deep because I have to nest all my children widgets inside a stupid children list. Another big problem with this approach is that my encapsulation for child management has been broken. I want to strictly control when children are added and removed because I have to maintain a doubly linked tree, dispose of native widgets, etc.

So that brings me to what I think my ideal model is:

EdgePane
{
  left = GridPane
  {
    numCols = 1
    Label { text = "alpha" }
    Label { text = "beta" }
  }
  right = GridPane
  {
    numCols = 2
    Label { text = "gamma" }
    Label { text = "delta" }
  }
}

This code is purely declarative so we can serialize easily to/from a file. Maybe not as tight as whitespace solutions like YAML, but about as concise as you can get for a {} language. But it won't currently work.

In my experience this notion of tree structured data where a given node has its own configuration and is a collection of children objects is extremely common. XML handles it various ways such as with attributes and elements. So how should Fan handle it?

My proposal is a feature I'm going to call unified collection literals. My primary focus here is how to solve the declarative modeling aspect. It has two sides: reading the model and writing the model.

If you use my example above it is fairly easy for the compiler to determine that Label { text = "xxx" } isn't an assignment to a field. When we encounter an expression which isn't itself a field assignment or method call on the with target, then we assume it an implicit call to add:

Team
{
  room = "101"
  Person { name = "alice" }
  Person { name = "bob" }
}

// above is syntax sugar for
Team
{
  room = "101"
  add(Person { name = "alice" })
  add(Person { name = "bob" })
}

// which compiles into
temp := Team.make
temp.room = "101"
temp.add(Person { name = "alice" })
temp.add(Person { name = "bob" })

I think that is a pretty simple, elegant solution to the reading problem - and it works for both InStream.readObj and the full Fan compiler.

So how does OutStream.writeObj know to output that same syntax? My proposal is a marker facet @collection which requires the class to implement an each method:

@serializable @collection
class Team
{
  Void each(|Person p|)
}

The OutStream.writeObj call would write all the normal fields as works today. Then if the @collection facet was specified it would also write all the objects returned by each.

That gives us a clean model for round trip serialization.

So let me take this a bit further:

// this
Team { Person { name = "bob }  }

// is syntax sugar for
Team { add(Person { name = "bob })  }

// then perhaps this
Team { "bob": Person {}  }

// is syntax sugar for
Team { add("bob", Person {})  }

Whoa! That is pretty much exactly what List and Map literals are doing. What we've effectively done is extended list and map literal syntax for use by any class that implements an add method. And we've created a clean way to mix configuration and children into one {} block without unnecessary nesting.

So if collection literals are now all the same, why use [] for lists and maps and {} for other types? If we unify all collection literals with [], that frees {} for use by closures and gets rid of the need for |,|! Yeah! My example written using []:

EdgePane
[
  left = GridPane
  [
    numCols = 1
    Label [ text = "alpha" ]
    Label [ text = "beta" ]
  ]
  right = GridPane
  [
    numCols = 2
    Label [ text = "gamma" ]
    Label [ text = "delta" ]
  ]
]

Looks a bit weird to me because I've been using {}, but it would be more internally consistent and frees expr {} for use with closures.

To summarize my proposal:

• serialization and with-blocks switch from {} to []
• support for implicit add inside with-blocks
• InStream.readOut switches to match with-blocks
• add @collection support to OutStream.writeOut
• remove requirement for |,| for no-arg closures

Comments?

Just thought of something... a complete switch to [] with-blocks won't work because [] is already used for indexing (get/set/slice shortcuts). Duh! But I feel like I'm onto something here - just have to figure out the right grammar.

I don't have the syntax answer, but why use @collection instead of a mixin (effectively interface here) called Collection with an abstract each method?

Hi Brian,

The only thing that concerns me a bit about this proposal is the magic call to add(). Why not make this method name explicit and configurable?

For example: a class could only be used in the context of a Unified Collection Literal if exactly one of its method has the facet @unified:

**
** Add the specified item to the end of the list.  The item will have
** an index of size.  Size is incremented by 1.  Return this.  Throw
** ReadonlyErr if readonly.
**
@unified
L add(V item)

-- Cedric

Less magic can only be a good thing, and as you've demonstrated, the more of the language you can implement in itself, the greater its expressive power. :)

As for the issue with [], I don't see it. If you're not using a scannerless parser, it just complicates the tokenizer a bit because it won't know whether use of [ represents indexing or addition. But there's no true ambiguity here because you can't index a class, so if an identifier is a class, then it represents addition.

I like the look of {} better than [] for building objects. Just aesthetics and familiarity for me. Maybe for others, too. Don't know, but familiarity can matter.

I like naming conventions (using add) more than facet (metadata) configurations. Not about magic so much as consistency. It goes along well with the operator naming style, too, I think.

Ah, but using {} just for closures has some advantages -- paving the way, for example, to see blocks following control structures as no-arg closures. Though I too prefer convention over configuration (in the case of add). It does match the operator naming style and leaves less guesswork for developers (they don't have to try to find the @unified facet, they can just look for add).

Perhaps {} after type names is a builder, and {} anywhere else is a closure. I think that would be OK. Besides, code inside builders can be arbitrary code so it's mostly a closure anyway. It would just be an implicit "with" callback. Other with uses (on existing objects) would need the keyword with.

I don't have the syntax answer, but why use @collection instead of a mixin (effectively interface here) called Collection with an abstract each method?

I think you already dug into this and saw the basic problem - the type system couldn't handle it very well. But serialization already uses an informal contract with duck typing for @simple. There isn't a Simple type for the same reason - there isn't a way to specify that a type must declare an static fromStr method.

The only thing that concerns me a bit about this proposal is the magic call to add(). Why not make this method name explicit and configurable?

I don't think it should be configurable because I think it is a good thing to force the method to be called add. In fact I've spent a lot of some trying really hard to be consistent with methods like size, get, set, add, etc across the entire library. Seems like consistency should trump configurability (in this case at least).

I like the look of {} better than [] for building objects. Just aesthetics and familiarity for me. Maybe for others, too. Don't know, but familiarity can matter.

After sleeping on it, I agree. Curly braces definitely feel more natural and the square brackets feel pretty forced.

Perhaps {} after type names is a builder, and {} anywhere else is a closure. I think that would be OK. Besides, code inside builders can be arbitrary code so it's mostly a closure anyway. It would just be an implicit "with" callback. Other with uses (on existing objects) would need the keyword with.

I started thinking about this exact design myself today. That starts to seem a fairly reasonable tradeoff. If most of the time with-blocks are with a constructor then we can figure out they aren't closures.

There doesn't seem to be any votes against the basic feature, so let's assume this design:

• with-blocks continue to use {}
• support for add and each for serialization
• list and map literals will continue to use []

If everyone agrees on that design, then we can debate this trade-off

Type {}   => with-block
|...| {}  => closure
expr {}   => with-block or closure?

Do we want to use a special syntax for no-arg closures or a special syntax for non-constructor with-blocks?

expr |,| {}   => special syntax for closure (currently required)
expr with {}  => special syntax for with-blocks

My thinking is that no arg closures are more important that non-constructor with-blocks, so it should get the better syntax. Does everyone agree with that? If so what do people think about for special with-block syntax:

expr with {}
expr.{}
any other ideas?

I'd say with(expr) {} (just like ECMAScript) or expr.{}.

By the way, there are risks of changing the meaning of code if the imported scope from with blocks can override outer local vars. That can happen when adding new items to the imported scope. Some people are willing to take that risk. I just want to make sure it's been considered.

Can you provide a code example for something which would be risky?

Say something like this, where in version 1 there is no name slot in TextField:

name := "Tom"
with (textField) {
  text = "Hello, $name!"
}

Then, in version 2, they introduce names on widgets, so there is a name slot in TextField. Then, rather than name referring to the local var, it refers to the imported slot and subtle unexpected results occur.

Technically, the issue exists for inner classes in Java, and I haven't seen it happen. Maybe not a big deal, but it seems like something that could very surprisingly break code, especially if there's no unit test covering the case. I think this is the kind of issue that motived this ES4 discussion on "reformed with".

Then, rather than name referring to the local var, it refers to the imported slot and subtle unexpected results occur.

With-blocks don't actually create a new scope in the classical sense. All with-blocks do is change the implicit target from this to the with-block target - but just for the base of the expression. So we should be safe from that error case.

So do unqualified references (lacking this.) default to local vars rather than the class members? If that's what you mean, then yes Fan is safe. I think either behavior could be "unexpected" to programmers, but the most stable choice semantically is to default always to lexical vars visible in the source file.

Identifiers resolve first to local variables in their scope, then class members. But with-blocks don't change what this means other than as the base:

// assumes no local variables (everything is class members)
goo = foo { bar(this, baz) }

// means
temp := this.foo()
temp.bar(this, this.baz)
this.goo = temp

In this case bar is evaluated against the base expression foo, but the rest of the expression uses normal scope resolution rules (those expressions bind against the declaring class, not the result of foo).

I've implemented the core of this feature:

• if the base expression of with-blocks supports an "add" method, then any expressions which don't resolve against the base expression are assumed to be base.add(expr)
• InStream.readObj will parse the new notation as calls to add
• OutStream.writeObj will serialize out the objects iteratered by each if type has @collection facet
• lots of new tests
• couple new sections in the documentation

I really like the way it turned out - although we are relying on the analysis phase to figure out what the AST really means, so that part is a little ugly. But it removes a lot of syntactic noise and gives us a clean way to do round-trip serialization of tree structures like widgets.

As I've worked through the feature, I don't really think this unifies perfectly with the list and map literals. Items within a with-block are separated by eos (newline or semicolon), while lists/maps are separated with comma. Plus to use a with-block you have to specify a base expression, whereas list/maps can typically use type inference. For example these produce the same result:

fansh> [2, 3, 4]
[2, 3, 4]
fansh> Int[,] { 2; 3; 4 }
[2, 3, 4]

Obviously I think convention should be to prefer the first, although the second falls out from having the general purpose with-blocks. I'm not sure we could really unify the two without running into all the same problems as we have with no-arg closures - there would have to be some special syntax to prefix the {} block of a list or map.

I did not change the grammar for with-blocks or closures - I will add a section to the roadmap doc as something we need to consider. I still would like to figure out how a no-arg closure is just {...}, but I'm having second thoughts about complicating the with-blocks syntax.

Good news above, by the way.

On the with subject, I find that Noun {...} looks like a builder (and with block) to me whereas verb {...} or adjective {...} looks like passing a closure, not a with block. My guess is that others familiar with Ruby, Groovy, or Scala (or perhaps even Java) will feel the same way. So, I think if people follow standard coding conventions on names that this is the right behavior.

Have you commonly used with blocks on existing objects rather than at construction time?

Have you commonly used with blocks on existing objects rather than at construction time?

Yes, I actually use them all the time with arbitrary expressions:

gc { color = Color.red; font = f; drawText("hi", 10, 10) }
list := Str[,] { capacity = n }
counts := Str:Int[:] { def = 0 }

What about

object.{...}    // with block
Class {...}     // builder
function {...}  // closure

?

I also got bitten by with blocks with code similar to this:

Bool f()
{
  return true
}

public Void main()
{
  f()
  {
    a = 0
    // do something with a
  }
}

gives:

/Users/cbeust/fan/a.fan(8,7): Unknown slot 'sys::Bool.a'

This is a habit I developed a long time ago and that I carried over with Java: whenever I have a need for a local variable for a scope that's much smaller than the method, I open a separate scope for it. It makes the code a bit easier to read and to move around (and for languages other than Java, it also helped the compiler generate better code).

Of course, this doesn't work in Fan, and in the example above, Fan attempts to set the slot called "a" to 0 on the object return by f(), which is a Bool.

Now, don't get me wrong: I think With Blocks are very promising and I'm happy to let go of my scoping habit to be able to use them, but this requires a small mindset shift.

-- Cedric

A keyword just for block scoping would be great. I use block scoping in Java, too, but it always looks so naked.

I have some thoughts on "with" and its relationship to the potential it feature, but I don't have time to type it up right now. I also don't have a full recommendation, just an apparently promising direction.

Per Helium's suggestion:

object.{...}    // with block
Class {...}     // builder
function {...}  // closure

The big problem is that I'd prefer to detect this in the parse phase because the AST is very different. The less we do at parse time, the more complicated and less powerful IDEs will be. But even if I figured it out after parse, you can't detect what is needed here:

Complex func(|,| closure)

func { ... }  // is this a closure or a with-block on the return of func?

A keyword just for block scoping would be great.

My Fan philosophy is that declarative programming is paramount. So given the choice I'm going to choose clean declarative programming (with-blocks) over a simplified closure syntax. Indeed the more code I write, the more I actually use with-blocks than closures.

Items within a with-block are separated by eos (newline or semicolon), while lists/maps are separated with comma. Plus to use a with-block you have to specify a base expression, whereas list/maps can typically use type inference.

It may be too late, but if you require type declarations for lists/maps, then unifying with-blocks and lists/maps is much simpler. Consider:

Str[,] list := ["a"; "b"; "c"]

which is interpreted as a with-block acting on a new list. Similarly:

Label label := [text = "hello"]

Equivalent to:

label := Label[text = "hello"]

I don't see why "," couldn't be used as the separator for with-blocks. Seems less cluttered than the semicolon character ;.

It's a semi-colon in with blocks because they can contain arbitrary code. I think that's also a reason why {} is better.

Concerning the "block scoping" I mentioned earlier, what I mean is just using different blocks for different scopes of local vars. This was in reference to cbeust's comments. My recommendation was to give a keyword just for this. For example, since do didn't fly for closures, it could perhaps be used here:

do {
  i := 1
  echo("i in first block: $i")
}
do {
  i := 2
  echo("i in second block: $i")
}

Here, do just means "run this now", sort of like saying do {...} while (false) or just plain {...} (as mentioned by cbeust) in other languages.

It's a semi-colon in with blocks because they can contain arbitrary code.

But as far as I can see, the comma character is used in exactly one place. Unambiguously, it can be either equivalent to the semicolon or used in place of it.