Btw, they had 3D pelican-on-the-bicycle easter egg in one of the promo videos: https://youtu.be/bOC3DisEOfg?t=117 so I'm pretty sure that they spent some small amount of resources to train the model to produce good svg version as well. :D

Your comparison grid is one of the most useful comparison grids I have ever seen in terms of measuring AI.

It feels silly to say that about making a pelican image but it really shows the difference in output and cost in an easy to understand way.

It seems to start understanding how bicycles work. Look at the front fork. There are reasons why it's shaped the way it is in real bicycles. If you do physical simulations in 3D with reinforcement learning you will start understanding that most of the arrangements that the other agents use just won't work.

You would not expect the developers of the model to optimize for a well known benchmark?

Simon do you have a page somewhere that shows _all_ of the penguins created by various models across time?

You’ve been doing this public service for so long (well, for so long in “AI hype” years anyway) that it’d be fascinating to see the evolution of this artifact across time.

I find curios that Astra's pelicans are basically the same (yellow sun top right corner, green bike, same bike shape, same legs style, very similar background) while in other there is more randomness.

Shouldnt you give the AI a different task each time? otherwise the model companies just optimise for this benchmark because its in their best interest to...

Astra pelican looks amazing: it looks like it was made by a professional artist. All the others look like they were made by kindergartners.

Do you ever randomize on a particular model - like try and get 3 outputs and pick one at random?

Coz who knows if astra low will produce max like output if tried once more.

They really can't decide which way pelican's knees should bend, do they?

It's mildly interesting that the bikes always seem to be front wheel facing the right.

Interesting that all of the bikes are turquoise colored, except for the medium effort

Oh my goodness. I was not prepared for Luna on "none".

Reminded me of https://clocks.brianmoore.com/

I love Astra pelicans! Awesome results!

Do you have other ideas of "combinations" for this kind of benchmark?

I'm wondering if this is being trained on by the models today.

Luna's price seems off

  Luna — costs in cents
  +--------+--------+---------+
  | Effort | Before | After   |
  +--------+--------+---------+
  | max    |   7.83 |    1.57 |
  | xhigh  |   4.24 |    0.85 |
  | high   |   2.46 |    0.49 |
  | medium |   1.26 |    0.25 |
  | low    |   0.76 |    0.15 |
  | none   |   0.71 |    0.14 |
  +--------+--------+---------+
  Per million tokens:
  Before: $1 input / $6 output
  After:  $0.20 input / $1.20 output

  Sol — costs in cents
  +--------+--------+---------+
  | Effort | Before | After   |
  +--------+--------+---------+
  | max    |  48.55 |   32.37 |
  | xhigh  |  24.11 |   16.08 |
  | high   |  10.38 |    6.92 |
  | medium |  10.55 |    7.03 |
  | low    |   8.33 |    5.55 |
  | none   |   5.90 |    3.93 |
  +--------+--------+---------+
  Per million tokens:
  Before: $5 input / $30 output
  After:  $4 input / $20 output

  Terra — costs in cents
  +--------+--------+---------+
  | Effort | Before | After   |
  +--------+--------+---------+
  | max    |  32.09 |   25.67 |
  | xhigh  |  14.67 |   11.74 |
  | high   |   3.74 |    2.99 |
  | medium |   3.46 |    2.77 |
  | low    |   3.47 |    2.78 |
  | none   |   2.60 |    2.08 |
  +--------+--------+---------+
  Per million tokens:
  Before: $2.50 input / $15 output
  After:  $2 input / $12 output

Crazy how much the Astra pelicans resemble GLM 5.3's (https://crimson-jeri-74.tiiny.site/), down to the color of the bike and the scarf.

The improved front fork design mentioned by Threatripper is about the only thing Astra is doing better, IMO.

You’re closer to this than I am but do you find it odd that helmets are almost never included? Biking is almost always accompanied by helmets

radial spokes on the back wheel is not a thing irl. i wonder if any model has gotten that right?

Luna is my go to daily model it's great value

Fallen off the homepage so what? I genuinely don't understand

[dead]

[dead]

[dead]

[flagged]

[flagged]

> That site build cost $24 - extremely non-trivial for a simple frontend.

I would say that $24 is trivial IF that's the final design. The truth is that the cost doesn't leave much room for error or experimentation.

I don't get it. For me it seems Opus was more accurate in terms of for example this small building in the right down corner

They look pretty similar to me, I'm not sure what you're seeing that is so much better.

At the bottom it says "Score 98.58", what measure is used for this score? It's kind of horrible, the perspective is all off (legs of the table makes that very obvious), the mouse/hamster has two mouths, a stub for a right paw, looks like left hand holds a melon on a stick or something, and there are pluses in the background for some reason. Not sure it'd call it "close to perfect" which the score seems to want to indicate.

That's quite shocking, at a sufficiently advanced level we can make all non-realistic graphics purely out of SVGs, as they'd have good scaling for things like logos and app icons. I know it was technically and theoretically possible before AI but most people weren't spending hours tweaking SVG HTML. I remember making an SVG dark mode toggle icon and it took days to get it right, I assume it's one shottable now.

The Astra one looks pretty good except it's standing on the wrong side of the table

>AGI

>Playing ping pong from the side of the table

I think marketing might be getting a bit absurd at this point

Not really comparable IMO. Astra and Fable are not the every day workhorse you reach for to do basic tasks (unless your company has fuck you-money), they’re the tool you break out when you need the absolute strongest performance. There are plenty of tasks where finding and fixing one or two extra edge cases saves the business a lot of money, even if the cost is high. The best example would be scanning for vulnerabilities, if these models weren’t kneecapped in that area.

the only thing that matters is cost per task. Astra seems to be massively efficient.

They gave out bankable resets for every day people on pro plans didn't get Astra. Given that, I wish they'd waited a few more days before activating it on my account :-D

(This applies to Sol but probably works for Astra too)

I've created with Sol a skill called Low Quota Mode that intends to reduce the use of tokens usages by the frontier (intelligent model) and delegate the use of bulk reading of docs/code and implementation to a sub-agent running Luna Max. Sol is asked to supervise, read the diffs and approves the commit/pr.

The skill might need some iterations while you use it, for example at the end of a rough session you can ask Sol how did it went, which were the points of conflict with Luna and try to iron them little by little by editing the skill.

Also in difficult tasks, ask to babysit the sub-agent model, I've seen it makes more effort into communication between frontier and sub-agent to guide the task with more care.

So far it has reduced my tokens usage a lot (have not quantified but the quota lasts more).

It is (uses way less tokens)

Same just got access literally this minute, Pro user here, no Cyber verification but have passed my ID over to them back in 2024 or something, maybe at the ChatGPT 3 API launch or something?

Has there been anything published about if Astra uses different amount of usage from your subscription plan compared to Sol? Don't recall coming across that in the press releases.

It cracks me up that you have to ask in a public forum because the vendors purposefully obscure this critical information.

The only reason most of my customers would use Azure Foundry instead of OpenAI directly is the ZDR assurance but it is so incredibly difficult to extract out of their model menu.

There is no trivial way to block non-ZDR models either so every customer has to “vet” and individually approve models.

If anyone from Microsoft is reading this: get your act together! You’re failing at the one thing people might want to pay you to do!

It's the same price as regular processing. You get guarantees microsoft give you, which are ones OpenAI won't (or require dedicated spend,) and you can use azure identities for access.

If you've got access to an Azure subscription and you don't pay the bill personally.

They’re ZDR and the OAI ones aren’t

Compared to OpenAI flex? I'm pretty sure that is their batch processing endpoint, which is naturally cheaper.

that's literally what the link is for...

You made an account just to post this?

That's over here: https://news.ycombinator.com/item?id=49570643