abhimanyu singh @siinghsaa

huggingface.co/blog/samuellim…

This is interesting as a first large diffusion-based LLM. Most of the LLMs you've been seeing are ~clones as far as the core modeling approach goes. They're all trained "autoregressively", i.e. predicting tokens from left to right. Diffusion is different - it doesn't go left to right, but all at once. You start with noise and gradually denoise into a token stream. Most of the image / video generation AI tools actually work this way and use Diffusion, not Autoregression. It's only text (and sometimes audio!) that have resisted. So it's been a bit of a mystery to me and many others why, for some reason, text prefers Autoregression, but images/videos prefer Diffusion. This turns out to be a fairly deep rabbit hole that has to do with the distribution of information and noise and our own perception of them, in these domains. If you look close enough, a lot of interesting connections emerge between the two as well. All that to say that this model has the potential to be different, and possibly showcase new, unique psychology, or new strengths and weaknesses. I encourage people to try it out!

Inception @_inception_ai

a year ago

We are excited to introduce Mercury, the first commercial-grade diffusion large language model (dLLM)! dLLMs push the frontier of intelligence and speed with parallel, coarse-to-fine text generation.

224 945 5K 1.9M 2K

@danish037 Funding or fellowship?

day 7 of #30DaysofFLCode started the @openminedorg courses finished L1/7 Introduction to Remote Data Science courses.openmined.org/courses/introd…

abhimanyu singh @siinghsaa

2 years ago

day 6 of #30DaysofFLCode yet another minicourse at DeepLearning.AI by Nicholas Lane and Daniel J. Beutel day 2/2 at Federated Fine-tuning of LLMs with Private Data finished

0 0 2 47 0

day 6 of #30DaysofFLCode yet another minicourse at DeepLearning.AI by Nicholas Lane and Daniel J. Beutel day 2/2 at Federated Fine-tuning of LLMs with Private Data finished

abhimanyu singh @siinghsaa

2 years ago

day 5 of hashtag#30DaysofFLCode yet another minicourse at DeepLearning.AI by Nicholas Lane and Daniel J. Beutel day 1/2 at Federated Fine-tuning of LLMs with Private Data lnkd.in/gwWiNGVQ the juiciest and longest chapter of the course yet to be finished

0 0 0 36 0

day 5 of hashtag#30DaysofFLCode yet another minicourse at DeepLearning.AI by Nicholas Lane and Daniel J. Beutel day 1/2 at Federated Fine-tuning of LLMs with Private Data lnkd.in/gwWiNGVQ the juiciest and longest chapter of the course yet to be finished

abhimanyu singh @siinghsaa

2 years ago

day 4 of #30DaysofFLCode day 2/2 finished the mini course just peeped into the SyftBox syftbox-documentation.openmined.org

0 0 2 37 0

day 4 of #30DaysofFLCode day 2/2 finished the mini course just peeped into the SyftBox syftbox-documentation.openmined.org

abhimanyu singh @siinghsaa

2 years ago

Day 3 of #30DaysOfFLCode day 1/2 mini-course by DeepLearning.AI on federated learning deeplearning.ai/short-courses/… authors have more emphasis on their framework libraries i.e. flower but still can get a rough idea. will try to replicate code on next day

0 0 2 90 0

Day 3 of #30DaysOfFLCode day 1/2 mini-course by DeepLearning.AI on federated learning deeplearning.ai/short-courses/… authors have more emphasis on their framework libraries i.e. flower but still can get a rough idea. will try to replicate code on next day

abhimanyu singh @siinghsaa

2 years ago

Day 2 #30DaysOfFLCode went through the video Intro to Privacy Preserving Artificial Intelligence - Andrew Trask youtube.com/watch?v=yUXwsN…

0 0 1 106 0

Day 2 #30DaysOfFLCode went through the video Intro to Privacy Preserving Artificial Intelligence - Andrew Trask youtube.com/watch?v=yUXwsN…

abhimanyu singh @siinghsaa

2 years ago

Day 1 of hashtag#30DaysOfFLCode went through Chapter 15, Intro to Federated Learning—Deep Learning on Unseen Data , of book Grokking Deep Learning by @iamtrask If you are a complete beginner in Deep Learning and federated learning please go through this chapter.

0 0 3 2K 2

Day 1 of hashtag#30DaysOfFLCode went through Chapter 15, Intro to Federated Learning—Deep Learning on Unseen Data , of book Grokking Deep Learning by @iamtrask If you are a complete beginner in Deep Learning and federated learning please go through this chapter.

abhimanyu singh @siinghsaa

2 years ago

I'm publicly committing to the #30DaysOfFLCode Challenge! (federated learning) Join me in learning more about #FederatedLearning → 30DaysOfFLCode.com

0 0 3 2K 0

I'm publicly committing to the #30DaysOfFLCode Challenge! (federated learning) Join me in learning more about #FederatedLearning → 30DaysOfFLCode.com

The upper bound for how long to pause AI is only a century, because “farming” (artificially selecting) higher-IQ humans could probably create competent IQ 200 safety researchers. It just takes C-sections to enable huge heads and medical science for other issues that come up.

📽️ New 4 hour (lol) video lecture on YouTube: "Let’s reproduce GPT-2 (124M)" youtu.be/l8pRSuU81PU The video ended up so long because it is... comprehensive: we start with empty file and end up with a GPT-2 (124M) model: - first we build the GPT-2 network - then we optimize it to train very fast - then we set up the training run optimization and hyperparameters by referencing GPT-2 and GPT-3 papers - then we bring up model evaluation, and - then cross our fingers and go to sleep. In the morning we look through the results and enjoy amusing model generations. Our "overnight" run even gets very close to the GPT-3 (124M) model. This video builds on the Zero To Hero series and at times references previous videos. You could also see this video as building my nanoGPT repo, which by the end is about 90% similar. Github. The associated GitHub repo contains the full commit history so you can step through all of the code changes in the video, step by step. github.com/karpathy/build… Chapters. On a high level Section 1 is building up the network, a lot of this might be review. Section 2 is making the training fast. Section 3 is setting up the run. Section 4 is the results. In more detail: 00:00:00 intro: Let’s reproduce GPT-2 (124M) 00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint 00:13:47 SECTION 1: implementing the GPT-2 nn.Module 00:28:08 loading the huggingface/GPT-2 parameters 00:31:00 implementing the forward pass to get logits 00:33:31 sampling init, prefix tokens, tokenization 00:37:02 sampling loop 00:41:47 sample, auto-detect the device 00:45:50 let’s train: data batches (B,T) → logits (B,T,C) 00:52:53 cross entropy loss 00:56:42 optimization loop: overfit a single batch 01:02:00 data loader lite 01:06:14 parameter sharing wte and lm_head 01:13:47 model initialization: std 0.02, residual init 01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms 01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms 01:39:38 float16, gradient scalers, bfloat16, 300ms 01:48:15 torch.compile, Python overhead, kernel fusion, 130ms 02:00:18 flash attention, 96ms 02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping 02:21:06 learning rate scheduler: warmup + cosine decay 02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms 02:34:09 gradient accumulation 02:46:52 distributed data parallel (DDP) 03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU) 03:23:10 validation data split, validation loss, sampling revive 03:28:23 evaluation: HellaSwag, starting the run 03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro 03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA 03:59:39 summary, phew, build-nanogpt github repo

“Are Transformers Effective for Time Series Forecasting?” represents a pivotal paper, decisively highlighting the shortcomings and deficiencies in research surrounding the use of transformers for #timeseries #forecasting. This paper effectively exposes the deceptive practices employed by various authors in their papers, such as inadequate benchmarking and other tactics, which have previously led to inflated claims regarding the performance of transformers in this domain.

#OWASP #Top10 for Large Language Model (#LLM) Applications v0.5.0 now out highlighting the top security & safety risks and issues that developers and security teams must consider when building applications leveraging #AI / LLMs: Download the PDF here👇 owasp.org/www-project-to…

How can you beat XGBoost, CatBoost, and TabNet on tabular data? Use a cocktail of 13 modern regularization techniques! (arxiv.org/abs/2106.11189) [1/9]

Sebastian Raschka @rasbt

4 years ago

A Short Chronology Of Deep Learning For Tabular Data: sebastianraschka.com/blog/2022/deep… Deep tabular methods are an interesting research direction! So, this morning, I sat down and summarized my thoughts + the recent papers I read.

42 193 910 0 458

All right, here is one trick for using XGBoost for *data analysis*. 1/5

javi santana @javisantana

4 years ago

@tunguz @simonw How do you use xgboost to understand the data?

6 0 21 0 1

Below are 7 popular Machine Learning algorithms implemented from scratch in Python🧵

This series of #Jupyter #Notebooks is a VERY nice step-by-step intro to data science and machine learning. If you're just starting out - I recommend walking through these notebooks as a first primer Definitely a great #100DaysOfMLCode project github.com/rasbt/python-m…

@mervenoyann Your notes are very crisp and are easy to follow ,love them