Max @Max__Web

Run LLMs locally in R with the rollama R package. The latest update simplifies query generation (Prompt) and adds support for Hugging Face models (@JohannesBGruber) Check it out: medium.com/@weber.aca/rol…

Lol, jfc. 1. People write a paper to critique a scale. 2. It gets 1250 cites 3. Turns out almost every one of those used the paper as SUPPORT for the scale link.springer.com/article/10.100…

Yea, compbio folks beware: Anaconda is now trying to get academic users to pay licensing fees. See this Reddit thread too: reddit.com/r/bioinformati…

Noah Reid @NM_Reid

2 years ago

uhh, anaconda just sent a message to our HPC admins that we're in violation their ToS and we now need to pay for a license or remove all their software from our system?

55 69 466 312K 187

For the few weeks, we've been working on developing a way to compare the efficiency of AI models on different tasks: we call it the Energy Star AI project! ⭐️⚡️ huggingface.co/blog/sasha/ene… Looking forward to hearing your thoughts and ideas - reach out if you want to collaborate🤗

This paper seems very interesting: say you train an LLM to play chess using only transcripts of games of players up to 1000 elo. Is it possible that the model plays better than 1000 elo? (i.e. "transcends" the training data performance?). It seems you get something from nothing, and some information theory arguments that this should be impossible were discussed in conversations I had in the past. But this paper shows this can happen: training on 1000 elo game transcripts and getting an LLM that plays at 1500! Further the authors connect to a clean theoretical framework for why: it's ensembling weak learners, where you get "something from nothing" by averaging the independent mistakes of multiple models. The paper argued that you need enough data diversity and careful temperature sampling for the transcendence to occur. I had been thinking along the same lines but didn't think of using chess as a clean measurable way to scientifically measure this. Fantastic work that I'll read I'll more depth.

Excited to announce the launch of #rollama, an #Rstats package that brings the power of generative LLMs to your R environment! Wrap the @ollama API for free, private, and reproducible use of genAI. Check out our (@JohannesBGruber) paper: arxiv.org/abs/2404.07654

💬1 week left to register for our Computational Social Science workshop on May 6th and 7th at Bielefeld University📢It features an amazing line-up of great speakers and we look forward to many inspiring discussions👉Registration details and programme here computational-social-science.org

@orioljbosch Good points. The first one can be resolved by using a seed with a model that can be downloaded. For example, using zephyr-7b-beta with a seed will give consistent outcomes.

🔑 Conclusions: 1. Importance of validation in text annotation with LLMs 2. Advantages of open models for data privacy 3. Reproducibility through downloadable models and predefined seeds

🧠 Key Insights: - Challenges of proprietary LLMs in social sciences - Advocacy for open models for better reproducibility & privacy - Case studies: Sentiment analysis in tweets & leisure activities in childhood essays - Performance evaluation of various open models

📢 New Preprint Alert! 🚀 🔍 "Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Models" by Maximilian Weber & Merle Reichardt 🔗 Read the full paper: arxiv.org/abs/2401.00284

The pace of open-source LLM innovation and research is breath-taking I suspect that open-source will soon become unbeatable for anyone except maybe OpenAI Here's why - Open-source community is way bigger than any specific company - Safety lobotomy and fear of bad press will continue will impact proprietary model performance - Smaller models that are instruct / fine-tuned are performing as well as 50x bigger models - Smaller models are more efficient and cheaper than large models - Companies will leverage open-source and offer value-added services and APIs

arxiv.org/abs/2310.06825 Mistral 7B paper is up on arxiv. The authorship order is alphabetical. Please cite with author = {Mistral AI} 🙂

📡The next term of the #TaDa Speaker Series starts next week!! We have six amazing speakers on all things #TextAsData/#NLP lined up for you 👇

How Can Open Source LLMs catch up to GPT-4V and Google's Gemini? Open-source LLMs are getting really good. However, they are not as powerful as GPT-4 right yet. Plus, mutlimodal models like GPT-4V and Google's Gemini will be dropping soon... making it even harder for open-source models to catch up to these closed APIs The trillion dollar question is, can open source models close the gap to closed source models? The answer is a cautiously optimistic - Probably! Here is how I see the open-source ecosystem evolving and catching up to SOTA closed source models over the next 12-18 months. Training multimodal open-source models - We at Abacus are working open-source multimodal models and I am hopeful that all other open-source research labs are doing the same thing. The good news is there is a clear path to building these and you can start with a pre-trained models like Llama-2 A common approach is to start with a pre-trained language model and then fine-tune it on multimodal tasks using a dataset that contains multiple types of data such as text, images, and/or audio. During this fine-tuning process, the model learns to relate information across different modalities, improving its performance on the target multimodal tasks. For instance, you might fine-tune a pre-trained language model using a dataset of images and associated captions to create a model that can generate descriptive text for new images it encounters. For example, Google used this technique by extending the initial PaLM language model. PaLM-E was enriched with sensor data from robots. This transitioned PaLM into a multimodal model, PaLM-E, capable of handling diverse tasks across robotics, visual, and language domains. Mimic-ing mixture of experts: Rumors suggest that GPT-4 may utilize a Mixture of Experts (MoE) architecture, where multiple smaller models, each specialized in different tasks, collaborate to process data. This setup allows the handling of a vast number of parameters more efficiently by distributing them across these "experts". It's speculated that such an architecture could help GPT-4 manage a more diverse range of tasks and data, scaling up its capacity and capability while controlling computational and memory demands. Mimic-ing this structure with open-source models is not hard. You can always instruct tune open-source models to be very good at a particular tasks and you then use multiple models each instruct tuned for a specific task to "collaborate" to answer queries. Leaderboards and benchmarks We already have a bustling open-source community with a number of LLM benchmarks including MT-bench where you can easily measure how your LLM compares to others. Open-source labs and developers are engaged in a constant race to beat SOTA open-source models. The community has already caught up to 3.5 and GPT-4 when fine-tuned for a particular task. The benchmarks are only going to get more robust as more and more open-source models are dropped New more powerful open-source models A number of companies including Meta, have committed to training next generation multimodal LLMs and open-sourcing them. Smarter AI alignment and RLHF - Open source models don't have as much scrutiny as big tech. This means that they don't need as much safety lobotomy as big tech closed-source APIs. The safety lobotomy has a harmful side effect of killing legitimate queries. For example Llama-2 refused to answer the query "how to kill a linux process" citing 'safety reasons'. Mistral-7B however answered that question correctly All this means that eventually open-source will likely catch up to closed source models. Over time, more efficient smaller models will match performance of larger models, reducing the need to have thousands of GPUs Already the Mistral-7B models beats the 13B Llama-2. We will continue to see such improvements on a on-going basis. At some point the law of diminishing returns will kick in for Google and OpenAI, unless there is a significant breakthrough in NNs and AI tech - i.e. just throwing more compute or data at the problem won't necessarily dramatically improve performance. This means that while open-source models may still be a year away from GPT-4, they will start closing the gap quickly!

LOL I guess this is happening

keyboard_arrow_left Previous keyboard_arrow_right Next

Researcher: Uses #crowdsourcing to evaluate LM output. Crowdworker: Uses #ChatGPT to produce labels. Researcher: Spends hours using #ChatGPT to figure out if crowdworker was cheating. Quo vadis, #NLProc ?

📢The Call for Participation for this year's summer school on the computational social science 💻of the democratic debate 🗨️🗯️🗳️is live now! Find it here: bigssscss.janlo.de (Deadline Apr 27). Note: A few more projects may appear in the coming week.

Surreal.