Benjamin F Spector @spectorb

math checks out! hadn't thought about it this way before your post but I think another way to view this is that you really just want to max the total shoreline of the system (provided enough low-latency networking). In your example, I think what the smaller-die approach is getting you is really shoreline/$, and allocating enough shoreline to networking allows the collective shoreline to behave as a coherent system.

1 0 22 4K 9

View Details

Benjamin F Spector @spectorb

3 months ago

@levidiamode @karpathy Thanks very much @levidiamode , this is so incredibly kind! Happy to do a follow up video with you if you'd like -- would be glad to talk a bit about Blackwell, megakernels, etc, if useful.

2 0 20 1K 3

View Details

Benjamin F Spector @spectorb

3 months ago

i propose, tonight only, we get to light off a nuke over the pacific. for america's 250th the world can let us have one special firework, just for the occasion

29 73 4K 136K 89

View Details

Benjamin F Spector @spectorb

3 months ago

- I saw the whole throughout-interactivity pareto frontier be characterized in real time on the A0 silicon, with the token outputs streamed too. - I went through their codebase for how they do megakernels and thought it was, genuinely, very well done. I actually agree it would be good if they put more of their numbers out in public. (FWIW I did encourage them to. I know why they haven't yet and it does make sense but I am not sure it's correct on net.) But I still dislike your insinuation that everyone around the table is a paid shill, which I think is insulting and also just wrong.

1 0 29 3K 3

View Details

Benjamin F Spector @spectorb

3 months ago

@__tinygrad__ @Etched This is stupid and you should feel bad. I have seen the actual hardware generating tokens at astonishing speeds. You must think very little of the people on this list to believe they would squander their reputations for a few dollars.

4 0 51 9K 3

View Details

Benjamin F Spector @spectorb

3 months ago

Had a chance to dive deep into this with the Etched team, and came away extremely impressed. It's a good chip sir. Their programming model around megakernels is also very simple and controllable and quite well done. Focus here on flop density + scale up seems right. Big kudos to the Etched team!

1 1 44 3K 5

View Details

Benjamin F Spector @spectorb

3 months ago

@InferizeAI Awesome stuff, can't wait to play with it!

0 0 4 666 0

View Details

Benjamin F Spector @spectorb

3 months ago

@EngramLab Congrats! Very excited to play with what you guys are building!

1 0 14 1K 0

View Details

Benjamin F Spector @spectorb

4 months ago

All reliable clusters are alike; each unreliable cluster is unreliable in its own way.

3 2 52 12K 7

View Details

Benjamin F Spector @spectorb

4 months ago

@hamzaelshafie What a wonderful, in-depth post! I'm immensely appreciative of the effort that must have gone into this. I've added a link to the readme of the TK repo. And footnote (2) is a good idea -- an xor instead of down shuffle would probably save a few instructions!

1 0 17 1K 1

View Details

will depue @willdepue

5 months ago

megakernels remain underrated. if you haven’t dug into them before go look them up! flappy seems to be hinting at some really powerful training megakernel stuff which is sick ex: fully contained training megakernel could be great for automated research

Flapping Airplanes @flappyairplanes

5 months ago

(4/5) One thing we’ve built is a “kittens” virtual machine that takes over the whole GPU and allows new kinds of co-optimization. We can go past the traditional sequential kernel model – for example, fusing entire training runs into a single kernel and even weirder stuff.

28 56 678 260K 305

7 19 513 61K 332

View Details

Benjamin F Spector @spectorb

5 months ago

Fun to share a little taste of some of the things we've been working on with @sequoia

Flapping Airplanes @flappyairplanes

5 months ago

(1/5) Great to be at @sequoia to give a sneak peek of one of our research directions! TL;DR one path to data-efficiency may be to “abuse GPUs like they’ve never been abused before”

14 74 1K 203K 815

2 3 54 12K 6

View Details

Hayden Prairie @hayden_prairie

5 months ago

We’ve been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters. Turns out the answer is YES – by looping blocks of layers during training. We find that predictable scaling laws exist for layer looping, allowing us to use looping to achieve the quality of a Transformer twice the size. Our scaling laws suggest that for a fixed parameter budget, data and looping should be increased in tandem! 🧵👇

42 179 1K 305K 1K

View Details

Benjamin F Spector @spectorb

7 months ago

@dylan522p @SemiAnalysis_ @Kurnalsalts Ah I meant they'd get physically larger to allow for lower power consumption. The whole thing has to run on like 100W or something? Including memory and whatnot. But this was an offhand comment, I am far from confident I am right.