GregoryKanevsky @DataGreg
recovering R-addict, dataholic, visual-maniac, so rather plain person novyden.blogspot.com Dallas, TX Joined September 2018-
Tweets261
-
Followers109
-
Following100
-
Likes382
Integrating RAGChecker metrics in Inspect-ai tasks linkedin.com/pulse/integrat… via @LinkedIn
Context window is one of key LLM features, but what vendors declare almost never the same as effective context length. Check out this graph showing @AI21's #Jamba's declared vs. effective #contextwindow
Hack by popular demand! Open-Source Agent #Hackathon with @AndrewYNg & @DeepLearningAI at @agihouse_org Saturday, *August 24th* Stellar schedule planned w/ secret guest speakers, alongside co-hosts, @GroqInc & @langchain. Reserve your spot now --> eu1.hubs.ly/H0bJRt70
📽️ New 4 hour (lol) video lecture on YouTube: "Let’s reproduce GPT-2 (124M)" youtu.be/l8pRSuU81PU The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model: - first we build the GPT-2 network - then we optimize it to train very fast - then we set up the training run optimization and hyperparameters by referencing GPT-2 and GPT-3 papers - then we bring up model evaluation, and - then cross our fingers and go to sleep. In the morning we look through the results and enjoy amusing model generations. Our "overnight" run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar. Github. The associated GitHub repo contains the full commit history so you can step through all of the code changes in the video, step by step. github.com/karpathy/build… Chapters. On a high level Section 1 is building up the network, a lot of this might be review. Section 2 is making the training fast. Section 3 is setting up the run. Section 4 is the results. In more detail: 00:00:00 intro: Let’s reproduce GPT-2 (124M) 00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint 00:13:47 SECTION 1: implementing the GPT-2 nn.Module 00:28:08 loading the huggingface/GPT-2 parameters 00:31:00 implementing the forward pass to get logits 00:33:31 sampling init, prefix tokens, tokenization 00:37:02 sampling loop 00:41:47 sample, auto-detect the device 00:45:50 let’s train: data batches (B,T) → logits (B,T,C) 00:52:53 cross entropy loss 00:56:42 optimization loop: overfit a single batch 01:02:00 data loader lite 01:06:14 parameter sharing wte and lm_head 01:13:47 model initialization: std 0.02, residual init 01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms 01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms 01:39:38 float16, gradient scalers, bfloat16, 300ms 01:48:15 torch.compile, Python overhead, kernel fusion, 130ms 02:00:18 flash attention, 96ms 02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping 02:21:06 learning rate scheduler: warmup + cosine decay 02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms 02:34:09 gradient accumulation 02:46:52 distributed data parallel (DDP) 03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU) 03:23:10 validation data split, validation loss, sampling revive 03:28:23 evaluation: HellaSwag, starting the run 03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro 03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA 03:59:39 summary, phew, build-nanogpt github repo
If you're a developer, this is probably the first chance you'll get to hack using a model that was developed with Mamba. Prizes include up to $5K in credits to build apps with Jamba on AI21 Studio (and lots of fun swag too!). Come meet the AI21 team and hack with us.
Come join @AI21Labs #Jambathon at the @AGIHouseSF on June 22nd to hack away with Jamba, the groundbreaking fusion of Mamba and Transformer architectures. agihouse-app.web.app/events/ai21-ja…
@ph_singer just like git never ceases to frustrate me, unix never ceases to amaze
Setup shell > echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc > echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc > echo 'eval "$(pyenv init -)"' >> ~/.zshrc Activate virtualenv automatically in your project > pyenv local [virtual-env-name]
Upgrade pip > pip install --upgrade pip Install Python > pyenv install -s 3.10.13 Remove virtual env > pyenv virtualenv-delete [virtual-env-name]
pyenv + virtualenv : > pyenv versions (once) > pyenv virtualenv <desired python version> <virtual env name> (once) > pyenv activate <virtual env name> (every time) > pip install -r requirements.txt (once) > pyenv deactivate (every time)
Git remove local branches after they were deleted in remote repo likely by closing PRs : > git fetch --all -p; git branch -vv | grep ": gone]" | awk '{ print $1 }' | xargs -n 1 git branch -d
@tunguz looks like someone got lazy and ChatGPT-ed this sticker 😀
Artificial Intelligence: The Good, The Bad and The Ugly - Free public lecture on May 24 by Professor Abu-Mostafa of @Caltech to explain the science of AI in plain language and assess extreme scenarios. work.caltech.edu/watson #MachineLearning #ArtificialIntelligence #AI #chatGPT
This is his last weekend with us
SunshineAmeliaMartin @Proutieg23978
35 Followers 2K Following Dare to be different Choose happiness every day
Tanee @TaneerZm5J
65 Followers 6K Following
vignesh kempannan @Vigneshkemp
5 Followers 183 Following
Ceerkear @CeerkearZkJC
47 Followers 3K Following
Raymond Peck @RaymondPeck14
0 Followers 18 Following
Jumping Rivers | @jum... @jumping_uk
7K Followers 2K Following #python, #rstats, #shiny, @mcmc_stan, #datascience training and consultancy. We help organisations extract the most from their data.
Danish Naseem @danishnaseem09
49 Followers 143 Following
Priya Ravindhran @PriyaR_AI
189 Followers 328 Following Entrepreneur/SaaS/Enterprise/AI Sales Leader, Dancer, Fitness Coach - Opinions are my own
Kim Montgomery @_dynamic24_
291 Followers 1K Following Applied Mathematician. Data Scientist. Kaggle Grandmaster.
The AI Academy @TheAIAcademy
20 Followers 64 Following High-Impact Boutique Consulting and Learning Services company. We help companies defining and executing their journey in the adoption of Artificial Intelligence
Michelle Tanco @MichelleTanco
69 Followers 75 Following 🌊 PM of the AI App Store at @h2oai 🎸 Bassist of @worsein 🌲 PNW native and certified tree hugger
Tahmina Afrin @TahminaAfrin9
75 Followers 549 Following I'm a professional graphic designer.I'm expert in illustration,line art,logo design,t-shirt .I love to do my job.
Niki Athanasiadou, Ph... @RodonikiA
629 Followers 690 Following I tweet about #AI #science #health #molecularbiology #research #innovation and sometimes #art -- all from #NYC . #Humor is important
Asghar Ghorbani @ghorbani_asghar
581 Followers 830 Following Working on https://t.co/B09fqY87UC currently, and few other things
Mateusz Dymczyk @mdymczyk
312 Followers 490 Following 🇵🇱🇫🇷🇯🇵🇬🇧 ML @Facebook Fighting bad guys @WhatsApp | ex-@H2Oai
Raymond E Peck III @raymondpeck3
516 Followers 2K Following 3 as in iii. Experienced, successful startup CTO/VP of Engineering/Director of Engineering/Principal Engineer + AI/Machine Learning/Data Science expert.
Mukharbek Organokov @circassia_ai
1K Followers 5K Following 🇫🇷 #AI | #Physics | #PhD | https://t.co/BgHc3uxsso ⋰Ẍ⋱ #Circassian #Адыгэ | https://t.co/1dcdRO3pnD Alumni: @unistra @Polytechnique @UnivParisSaclay @pgpuspb @SPbGU
Click2Refund | Client... @Click2refundC
12 Followers 341 Following Client Relations of Click2Refund. Click2Refund protects your rights as passengers! We claim compensation for your delayed and canceled flights.
Rohan Rao @magras193
36K Followers 609 Following 💛 ML @h2oai 👨💻 Quadruple Kaggle Grandmaster 🏆 x 9-time Indian Sudoku Champion 🇮🇳 🥈 Asian Championship 🏆 You v YouTube winner 🎓 IIT-Bombay
Vinod Iyengar @VinodIyengar
641 Followers 498 Following Product Executive | @ThirdAILab, ODA 5, Angel Investor Previously early @h2oai @earnin, @rushcard I write about AI, Product Strategy, B2B SaaS
Pritam Raj @PritamR77468418
9 Followers 114 Following
Quantzig @Quantzig
667 Followers 890 Following We are an advisory firm that specializes in global analytics. We assist our clients with leveraging analytics for prudent decision making.
Jan Gamec @JanGamec
89 Followers 191 Following @h2oai New technologies. Machine Learning. Programming. Cryptography. Violin. Nature
Pramit Choudhary @MaverickPramit
605 Followers 994 Following 🏂 Started OidLabs, ex-Lead ML Scientist(Engineer) @h2o.ai UCI alumni Currently exploring possibilities of LLMs. Personal views only, not employers
@[email protected]... @AlanJumpi
503 Followers 337 Following I'm a BNFH (Bastard Nerdbanger From Hell) !!
Gurjarji @MathsOmiitd
0 Followers 10 Following
Karthik Reddy @akreddy74
24 Followers 150 Following
Sirisha Sri @SiriBavireddy
55 Followers 138 Following
Yash ✨ @yashpathack
219 Followers 2K Following
Adriana Tomic @TomicAdriana
1K Followers 2K Following Systems immunologist using #MachineLearning #AI to understand human immunity to #viruses #vaccines
Crispin Sujith @crispinsujith27
5 Followers 160 Following
Tanner Stauss @Tanner_Stauss
142 Followers 794 Following Data Science Engineering || R Programmer || Shiny Developer #rstats
Michel Sebag @MichelSebag
279 Followers 5K Following Group Head of AI & Machine Learning #DataScience #MachineLearning #DeepLearning #BigData #ArtificialIntelligence #NLP #BayesianNets (opinions are mine)
Demis Hassabis @demishassabis
1.3M Followers 175 Following Nobel Laureate. Co-Founder & CEO @GoogleDeepMind - working on AGI. Solving disease @IsomorphicLabs. Trying to understand the fundamental nature of reality.
AI21 Labs @AI21Labs
11K Followers 162 Following AI21 Labs builds Foundation Models and AI Systems for the enterprise that accelerate the use of GenAI in production. Meet AI21 Maestro https://t.co/IJyxlWYJoV
Percy Liang @percyliang
108K Followers 425 Following professor of computer science @Stanford @stanfordnlp, co-founder of @togethercompute, creator of https://t.co/7R5THVogW2, co-founder of @simile_ai, pianist
Daniel Gross @danielgross
240K Followers 0 Following
Alex Graveley @alexgraveley
46K Followers 2K Following Co-creator Perplexity Computer, GitHub Copilot, Dropbox Paper. 2x CEO. Thruhiker. Survivor 🎗️
Naveen Rao @NaveenGRao
38K Followers 951 Following CEO @unconvai. Former CEO MosaicML/Databricks & Nervana/IntelAI. Neuro + CS. I like to build stuff that will eventually learn how to build other stuff.
Riley Goodside @goodside
214K Followers 3K Following Screenshots of chatbots since 2022. Formerly: Google DeepMind, Scale
AGI House SF @AGIHouseSF
12K Followers 305 Following In the golden age of machine learning we're bringing hackathon life back to Silicon Valley! Shaping the future of AI, one line of code at a time.
Sebastián Ramírez @tiangolo
84K Followers 279 Following Creator of @FastAPI, Typer, SQLModel, Asyncer, etc. 🚀 From 🇨🇴 in 🇩🇪 . Open Source, APIs, and tools for data/ML. 🤖 Building @FastAPIcloud. ⚡️
anton @abacaj
48K Followers 606 Following
John Kinson @johnkinson
508 Followers 342 Following CTO and GTM Advisor GenAI, streaming data, cloud architecture, music production, travel, and Spanish culture
IMDb @IMDb
4.3M Followers 1K Following Helping you identify “that person from that one movie” since 1990. 🔍
Sebastian Raschka @rasbt
467K Followers 1K Following ML/AI research engineer. Ex stats professor. Author of "Build a Large Language Model From Scratch" (https://t.co/O8LAAMRzzW) & reasoning (https://t.co/5TueQKx2Fk)
Mark Landry @Mark_a_Landry
338 Followers 21 Following Direct of data science & product at H2O; Kaggle grandmaster; father of two.
Yauhen Babakhin @ybabakhin
324 Followers 76 Following Applied Scientist @nvidia | Kaggle Grandmaster https://t.co/u32pwLHjY4
RelationalAI @RelationalAI
1K Followers 112 Following RelationalAI brings enterprise decision intelligence to Snowflake's AI Data Cloud.
Kim Montgomery @_dynamic24_
291 Followers 1K Following Applied Mathematician. Data Scientist. Kaggle Grandmaster.
Michelle Tanco @MichelleTanco
69 Followers 75 Following 🌊 PM of the AI App Store at @h2oai 🎸 Bassist of @worsein 🌲 PNW native and certified tree hugger
Niki Athanasiadou, Ph... @RodonikiA
629 Followers 690 Following I tweet about #AI #science #health #molecularbiology #research #innovation and sometimes #art -- all from #NYC . #Humor is important
Priya Ravindhran @PriyaR_AI
189 Followers 328 Following Entrepreneur/SaaS/Enterprise/AI Sales Leader, Dancer, Fitness Coach - Opinions are my own
Hopsworks @hopsworks
1K Followers 318 Following Overcome legacy systems with a seamless, modular and performance-driven AI Lakehouse. Build, deploy and manage models effortlessly. https://t.co/2R2TqyW1qP
Tecton @TectonAI
1K Followers 112 Following Build and deploy production-grade machine learning applications with the #FeaturePlatform for #MachineLearning, from the creators of Uber Michelangelo.
Iguazio (Acquired by ... @iguazio
929 Followers 1K Following Implement and Scale your ML and Gen AI Applications
David Karger (hci.soc... @karger
4K Followers 147 Following Professor of CS at MIT Moving to Mastodon, user @karger at server hci dot social
Philipp Singer @ph_singer
12K Followers 496 Following Founding Data Scientist @prior_labs | PhD in CS Top ranked Kaggle Grandmaster (Highest #1) All views are my own. https://t.co/NHdaca1Nns
Global Fish @sigmoid92
70 Followers 350 Following #tech #photo #motorcycle #iot #aiml #hobbyelectronics #funnyguy making a hash of it all
Rohan Rao @magras193
36K Followers 609 Following 💛 ML @h2oai 👨💻 Quadruple Kaggle Grandmaster 🏆 x 9-time Indian Sudoku Champion 🇮🇳 🥈 Asian Championship 🏆 You v YouTube winner 🎓 IIT-Bombay
Ryan Reynolds @VancityReynolds
20.6M Followers 2K Following owner: @aviationgin - @MintMobile - @maximumeffort - @Wrexham_AFC
Dmitry Larko @DmitryLarko
803 Followers 58 Following
Pedro Domingos @pmddomingos
131K Followers 178 Following Professor of computer science at UW and author of '2040' and 'The Master Algorithm'. Into machine learning, AI, and anything that makes me curious.
Vinod Iyengar @VinodIyengar
641 Followers 498 Following Product Executive | @ThirdAILab, ODA 5, Angel Investor Previously early @h2oai @earnin, @rushcard I write about AI, Product Strategy, B2B SaaS
Sanyam Bhutani @bhutanisanyam1
42K Followers 1K Following 👨💻 Working on llama models @AIatMeta | Previously: @h2oai, @weights_biases 🎙 Podcast @ctdsshow 👨🎓 Fellow @fastdotai 🎲 Grandmaster @Kaggle
@[email protected]... @AlanJumpi
503 Followers 337 Following I'm a BNFH (Bastard Nerdbanger From Hell) !!
Badr C. @badrchentouf
144 Followers 80 Following
Prithvi @CrunchingData
294 Followers 64 Following Chief of Tech @h2oai. Graphics, visualization, compilers. Making app dev fast, fun and easy. Try https://t.co/dMXmDhygVs and https://t.co/dsswdwnC7n
Pramit Choudhary @MaverickPramit
605 Followers 994 Following 🏂 Started OidLabs, ex-Lead ML Scientist(Engineer) @h2o.ai UCI alumni Currently exploring possibilities of LLMs. Personal views only, not employers
Sirisha Sri @SiriBavireddy
55 Followers 138 Following
Jan Gamec @JanGamec
89 Followers 191 Following @h2oai New technologies. Machine Learning. Programming. Cryptography. Violin. Nature
PS @pstetsenko
108 Followers 172 Following
Ryan Chesler @ryan_chesler
2K Followers 483 Following Applied Research Scientist @Nvidia. Kaggle double grandmaster and organizer of the San Diego Machine Learning meetup





















