#557 object 'as' type: is this a bug or am I doing it wrong?

KevinKelley Wed 29 Apr 2009

This example code:

mixin Tree
{
  abstract Tree? parentNode
  abstract Tree[]? childNodes

  virtual Void addChild(Tree child)
  {
    child.parentNode = this
    if (childNodes == null) childNodes = [,]
    childNodes.add(child)
  }
  //...
}

class MyTree : Tree
{
  override Tree? parentNode
  override Tree[]? childNodes

  MyTree? parent { get { return parentNode as MyTree? } }
  MyTree[]? children { get { return childNodes as MyTree[]? } }


  static Void main()
  {
    MyTree root := MyTree()
    root.addChild(MyTree())

    echo(root.childNodes)

    echo(root.children)

  }
}

generates this output:

[fan.bug2_0.MyTree@47393f]
null

and I don't know why. I'm trying to make an accessor for the mixin's data fields that can be used without casting. But, I'm getting null back from the children accessor method.

If I change the accessor to:

MyTree[]? children { get { return (MyTree[]?)childNodes  } }

then I get the output I want:

[fan.bug2_0.MyTree@47393f]
[fan.bug2_0.MyTree@47393f]

So, no big deal. But I don't get why the as isn't doing what I thought it would.

brian Wed 29 Apr 2009

If you dump the type of childNodes you will see it is sys::Obj?[] since it is allocated with the expression [,]. Since Obj[] is not a subclass of MyTree[] so it is the correct behavior to return null. The thing here is that the type of the list is what matters, not necessarily what is inside it.

Without knowing exactly what you are doing, I tend to tackle these sort of problems with covariance like this:

abstract class Base
{
  abstract Base parent()
}

class MyBase : Base
{
  override MyBase parent
}

Side note: using a nullable type for the as expression is implied, I am going to make that a compiler error.

KevinKelley Wed 29 Apr 2009

Okay... so when I used the as it was giving null to report that the array isn't of that type. But when I used the cast operator it was coercing the array to the type, and since it only held elements of the correct type, the cast worked. Correct?

I had been thinking that as was just another syntax for the cast operator.

Btw the example above just comes from making a covariant accessor for the non-covariant mixin field.

JohnDG Wed 29 Apr 2009

Okay... so when I used the as it was giving null to report that the array isn't of that type. But when I used the cast operator it was coercing the array to the type, and since it only held elements of the correct type, the cast worked.

This behavior surprises me. I would expect as to return null only in the cast that casting would throw a cast exception.

brian Wed 29 Apr 2009

I changed the compiler to not allow a nullable type to be used with the as expression.

Okay... so when I used the as it was giving null to report that the array isn't of that type. But when I used the cast operator it was coercing the array to the type, and since it only held elements of the correct type, the cast worked. Correct?

For most types, things work like this:

// Fan code      emitted Java code
(Foo)x           (Foo)x
x as Foo         x instanceof Foo ? (Foo)x : null

But Lists, Maps, and Funcs work a little different since the Java type system doesn't actually capture their types:

A cast to List/Map/Func type is basically a no-op:

(Str[])x

In the example, we don't actually walk the list to check each item. That kind of sucks because you might get a type error when you actually try to use the list later. But given that Fan does a lot of auto-casting, we definitely don't want to be walking the collection for type checks.

In the case of is and as operator for generics, we actually generate this code:

x is Str[]            // Fan code
x.type.fits(Str[]#)   // what really gets generated

Hope that helps explain what is really going on under the covers.

tompalmer Wed 29 Apr 2009

I still expect a CastErr for normal casting if as returns a null. If it can be determined in one case, why not the other?

brian Wed 29 Apr 2009

I still expect a CastErr for normal casting if as returns a null. If it can be determined in one case, why not the other?

Using the current design (where a generic cast is basically a no-op), then the matching behavior for as with a generic type is to evaluate never to null (assuming correct base type of List, Map, or Func).

That makes sense to me for consistency. If there is consensus around that behavior I will change it.

JohnDG Wed 29 Apr 2009

In the example, we don't actually walk the list to check each item. That kind of sucks because you might get a type error when you actually try to use the list later.

I wonder if it would be feasible to store the type of the list in the list itself, e.g., list.elemType, in which case add could check that the new element fits the element type, and the compiler could do runtime checking of casting from one type of list to another type.

In any case, like Tom, I expect a CastErr in the exact case that as returns null. It's really surprising this isn't how Fan works, so if you want to maintain the existing behavior, I suggest making it crystal clear in the documentation for as and casting.

brian Wed 29 Apr 2009

I wonder if it would be feasible to store the type of the list in the list itself, e.g., list.elemType, in which case add could check that the new element fits the element type, and the compiler could do runtime checking of casting from one type of list to another type.

The List does know its value type. But it seems really expensive to check every add, and in three years of programming in Fan that has actually never caused an error that I've seen.

But here is the tricky problem:

objs := Obj[,]
objs.add("a").add("b")
strs := (Str[])objs

Today, strs === objs, and would still report Obj[] as its type. We could potentially walk objs, check that everything is Str and return a new list typed as Str[]. But that seems awfully expensive.

In any case, like Tom, I expect a CastErr in the exact case that as returns null.

Yeah, I agree with you and Tom that the current behavior is inconsistent and should really be changed.

KevinKelley Wed 29 Apr 2009

I agree, current behavior is surprising, and I think as and cast should be consistent with each other: CastErr when as returns null, seems right.

If List is really always Obj[], then it should be always safe to cast it to SomeType[], since SomeType always derives from Obj and can therefore be put in any list. Allowing as SomeType[] lets me use the compiler to ensure that the list only holds what I want it to. So, List as AnyType[] should always be non-null, I'm thinking.

qualidafial Wed 29 Apr 2009

This rule does seem right except when the variable being casted is null.

Obj? x := null
Str? y := x        // x is null, so casting is legal in this case
Str? z := x as Str

So while as returns null whenever a cast would throw a CastErr, the reverse scenario is not necessarily true.

JohnDG Wed 29 Apr 2009

But it seems really expensive to check every add

In Java this is an identity comparison; e.g. x.class == elemType.class, and therefore quite fast (as fast as two dereferences and a conditional can be).

We could potentially walk objs, check that everything is Str and return a new list typed as Str[]. But that seems awfully expensive.

I like the check for all elements, because it locates any potential problems to where the cast is first performed -- as opposed to 10 function calls later, when an object from such an array is used in a way incompatible with its type.

However, you couldn't create a new list, because then changes to the original would not be reflected in the new list, which would break the semantics of casting.

KevinKelley Wed 29 Apr 2009

So, List as AnyType[] should always be non-null, I'm thinking.

On second thought, after seeing Brian's example, maybe that doesn't make as much sense.

objs := Obj[,]
objs.add("a").add(1)     // okay
strs := objs as Str[]    // now what?  

JohnDG Wed 29 Apr 2009

CastErr ideally.

brian Wed 29 Apr 2009

In Java this is an identity comparison; e.g. x.class == elemType.class, and therefore quite fast (as fast as two dereferences and a conditional can be).

It isn't quite that simple. For example a list of Num can store Int, Float, or Decimal - I follow Java's contra-variant array model here.

So we have to check subclasses. In many cases we can utilize Class.isAssignableFrom which I understand HotSpot optimizes to be near the performance of a instanceof operation. However since we can't use the Java type system to capture generic types like Int:Str, in the end we would have to route to Type.fits which will always be pretty slow because I have to walk the inheritance tree myself in Java/C# code.

I don't really love the current solution, but I do think it utilizes the best trade-off of compile time safety with runtime performance. Basically Fan isn't much different than what type-erased-generics are doing in Java. Although I think Fan is better because at least the generic types maintain their parametrization for reflection (even if they don't use them for runtime type checking).

brian Wed 29 Apr 2009

@KevinKelley:

objs := Obj[,]
objs.add("a").add(1)     // okay
strs := objs as Str[]    // now what?  

As JohnDG said, ideally it would throw a CastErr. But practically speaking, as a semi-dynamic language, there isn't much we can do until someone attempts to get an item as a Str and gets a CastErr. That isn't ideal because the exception doesn't occur where the actual problem is. But I still believe it is the right trade-off because that sort of error is extremely rare, where auto-casting Lists/Maps is extremely common.

qualidafial Wed 29 Apr 2009

objs := Obj[,]
objs.add("a").add(1)     // okay
strs := objs as Str[]    // now what?  

This may be a stupid idea but I thought I'd throw it out there: how about wrapping the target of the as expression in a special list that simulates the as semantics on each element during traversal?

class AsList : List
{
  V[] target
  Type of

  AsList(V[] target, Type of)
  {
    this.target = target
    this.of = of
  }

  Void each (|V?, Int| func)
  {
    target.each |val, i| {
      func(of.fits(val) ? val : null, i)
    }
  }

  ...
}

qualidafial Wed 29 Apr 2009

On second thought, that would weaken the result type of the as expression. That is, the expression strs := objs as Str[] would yield a Str?[]? instead of the expected Str[]?.

brian Wed 29 Apr 2009

This may be a stupid idea but I thought I'd throw it out there: how about wrapping the target

As JohnDB pointed out the result of an as expression must reference the original instance. That is this axiom must apply:

r := x as T
if r != null then r === x must be true

tompalmer Wed 29 Apr 2009

That is this axiom must apply ...

Unless it's an Int, Float, or Bool being unboxed? Well, I guess if it was an Obj (or Num) before, and it stayed boxed (since as gives nullable results), maybe not an issue here.

Just thinking about where === expectations might break. I guess normal casting (to plain Int rather than Int?, say) could break them.

brian Wed 29 Apr 2009

Just thinking about where === expectations might break. I guess normal casting (to plain Int rather than Int?, say) could break them.

The compiler will report an error if you attempt to use the as operator with a value type (so that case is already covered).

brian Thu 30 Apr 2009

Is everyone comfortable with the trade-offs I have made regarding casts of generic types? If the compiler knows that the cast will fail it will report an error - for example casting Str[] to Int[] will never be allowed. Otherwise if the cast could succeed the compiler auto-casts, but the runtime does not check it.

I haven't done a lot of work with Java generics, but I assume they work the same way.

Assuming we agree to that behavior, then I propose to change the as operator work consistently with casting for generic types. Since the cast operator always succeeds with a generic type (assuming the correct base type), then the as operator will always return non-null.

KevinKelley Thu 30 Apr 2009

I agree; that seems best.

tompalmer Thu 30 Apr 2009

I'd lean towards making it fail in both cases (if it's not a huge performance issue), but I think consistency is better than not, so I agree that allowing it to succeed in both cases is better than the current behavior.

brian Thu 30 Apr 2009

I pushed a fix to make as work like cast. Note that is still actually calls Type.fits, but I think that is that the correct behavior:

fansh> Obj o := Obj["a", 3]
[a, 3]
fansh> o as Str[]
[a, 3]
fansh> o is Str[]
false
fansh> (Str[])o
[a, 3]

KevinKelley Thu 30 Apr 2009

Seems right to me: can use strongly-typed lists if I want, or use them as loosely-typed collection of Obj if I want.

alexlamsl Thu 30 Apr 2009

wait - does that mean I would be expecting an Int from a Str[]?

brian Thu 30 Apr 2009

Here is parallel Java code for this problem that pretty much works just like Fan does:

ArrayList list = new ArrayList();
list.add("foo");
ArrayList<Integer> x = (ArrayList<Integer>)list;  // this works
System.out.println(x.get(0).intValue());          // fails here

Although I notice that Java won't let you cast from List<Object> to List<Integer>.

JohnDG Thu 30 Apr 2009

Although I notice that Java won't let you cast from List<Object> to List<Integer>.

This is actually a major annoyance, because the Java type system is not powerful enough to express some relationships, which means the ultra conservative generics rules often just get in the way and prevent perfectly good code from compiling.

qualidafial Thu 30 Apr 2009

Could we have a method List.checked() so that we have the option to choose type safety over performance?

brian Thu 30 Apr 2009

Could we have a method List.checked() so that we have the option to choose type safety over performance?

I don't mind adding a new method to List, but it will have to be something you call explicitly (the compiler won't call it).

Today you could do something like:

list.all { it.type.fits(Expected#) }
list.findType(Expected#).size == list.size

We could add something like this which returns true if all the items fit a specific type:

Bool allType(Type t)

JohnDG Thu 30 Apr 2009

Could we have a method List.checked()...

Personally I'd prefer a SafeList that checks on addition, because it's most helpful to get the error at the point of insertion, rather than much later when a client decides to call a checked method.

qualidafial Thu 30 Apr 2009

I don't mind adding a new method to List, but it will have to be something you call explicitly (the compiler won't call it).

Agreed.

We could add something like this which returns true if all the items fit a specific type:

I'm not sure we need to go that far. Just wrap the list in an instance that checks element type on insertion. This is all that java.util.Collections.checkedList() does--it does not verify the elements already present in the list. If you're really worried about type safety you can always call checked() at the time the list is constructed.

The two cases I see for using this API are:

  1. Some buggy code somewhere in the program is adding an element of the wrong type. A bug like this is generally hard to track down because the error doesn't surface until later when you try to cast the bad element to the expected type. In this scenario checked is used to make the error surface immediately.
  2. You are passing a List to some outside code that is going to change the list's contents, and you want to guard against the outside code doing bad things to your list.

I'm sure there are other uses but these are the two main ones I can think of.

In the same vein I think we could also add Map.checked() (as well as List.isChecked() and Map.isChecked()).

qualidafial Thu 30 Apr 2009

Personally I'd prefer a SafeList that checks on addition, because it's most helpful to get the error at the point of insertion, rather than much later when a client decides to call a checked method.

I think we're on the same page, except I'm calling it "checked" and you're calling it "safe."

Just to clarify, what I'm suggesting is similar in semantics to List.ro() except that you're getting a type-checking view instead of a read-only view.

brian Fri 1 May 2009

@qualidafial - do you mind creating a new topic to propose and discuss this feature?

qualidafial Fri 1 May 2009

alexlamsl Fri 1 May 2009

Although I notice that Java won't let you cast from List<Object> to List<Integer>.

Yes, I forgot - Obj as type parameter in Fan is the Java equivalent of raw type.

It does get a bit of time and practice to get used to, but once you are there it is actually quite natural.

I'm thinking of some sort of compiler warning here (like the unchecked warnings that you get for casting List to List<Integer>), but then the dynamic nature of Fan won't make much sense this way...

All in all, I'm happy with where things are right now :-)

P.S. List<Object> is not a raw type, so the cast will fail.

Login or Signup to reply.