DuckAI @TheDuckAI

An open-source ML research community at Discord: https://t.co/7YDTo6Mo1G duckai.org Joined October 2022

Tweets

23
Followers

663
Following

8
Likes

35

Enrico Shippole @EnricoShippole

2 years ago

@TeraflopAI is excited to help support the @caselawaccess and @HarvardLIL, in the release of over 6.6 million state and federal court decisions published throughout U.S. history.

2 36 95 30K 70

View Details

Aran Komatsuzaki @arankomatsuzaki

2 years ago

@TeraflopAI is excited to help support the @caselawaccess and @HarvardLIL, in the release of over 6.6 million state and federal court decisions published throughout U.S. history 🥳

1 7 34 5K 11

View Details

DuckAI @TheDuckAI

3 years ago

@softyoda We simply use OBS for video recording, and I'm afraid I do not know what kind of codec compression OBS uses. You can probably find out in their repo: github.com/obsproject/obs…

0 0 2 69 0

View Details

🌟 Introducing DuckTrack 🦆 - Multimodal Computer Interaction Data Collector - High-accuracy tracking & playback - Synced collection of mouse, keyboard, screen video & audio data - 💻 Desktop app for major OSs. Explore more 👇 duckai.org/blog/ducktrack github.com/TheDuckAI/Duck… 1/N

6 27 145 78K 124

View Details

DuckAI @TheDuckAI

3 years ago

@WonderinCaveMan Hi Leon. The amount it takes is comparable to OBS, which is small enough that it does not noticeably slow down your computer :)

0 0 0 94 0

View Details

DuckAI @TheDuckAI

3 years ago

@OmarKamali Yeah that's on our TODO list :)

1 0 1 32 0

View Details

DuckAI @TheDuckAI

3 years ago

@tensorcel Thanks for your feedback! Yeah. That's something we've discussed that need to be done :) We're looking for contributors to add more features or fix the said limitations

0 0 1 358 0

View Details

DuckAI @TheDuckAI

3 years ago

@OmarKamali Given that some processes are not deterministic, the use of playback for other purposes may not make sense. But if anyone finds any other use case, that'd be awesome :)

0 0 0 10 0

View Details

DuckAI @TheDuckAI

3 years ago

@OmarKamali The primary purpose of the playback feature is that (1) it can be easily modified to allow real-time conversion of computer agent model outputs to computer actions, (2) we can test the accuracy of the recording and various misc. stuff

2 0 1 32 0

View Details

DuckAI @TheDuckAI

3 years ago

🤝 Join Our Mission to Evolve DuckTrack 🛠️ Current Focus: - Replacing 'pynput' with 'libuiohook' for enhanced functionality. - Further improve accuracy - Improving stability and usability Work done by @broyojo42 and @arankomatsuzaki 5/N

1 1 8 1K 0

View Details

DuckAI @TheDuckAI

3 years ago

🌍 Community Data Collection Initiative for DuckTrack - We're inviting the open-source community to participate in an effort to collect a diverse range of computer interaction data. - Details on how to contribute and the data collection setup will be announced soon! 4/N

1 0 7 1K 0

View Details

DuckAI @TheDuckAI

3 years ago

Please see github repo readme for updated instructions on accessing API. (In particular, the instructions at the bottom of page 14 in the preprint is outdated.) github.com/theduckai/arb

0 1 4 2K 2

View Details

DuckAI @TheDuckAI

3 years ago

@sarahcat21 @duckdb Thanks for reminding me to add @duckdb to the list of companies to collaborate, which includes @DuckDuckGo and @aflac.

1 0 1 236 0

View Details

DuckAI @TheDuckAI

3 years ago

We are releasing the Advanced Reasoning Benchmark dataset for LLMs (ARB)! - Evaluates SotA LLMs on ARB, on which even GPT4 struggles - Explores the feasibility of letting LLMs generate and use rubrics to evaluate generated solutions. proj: arb.duckai.org 👇🧵(1/N)

5 48 266 97K 155

View Details

DuckAI @TheDuckAI

3 years ago

@LearningReason On a related note, we observed that, if we feed the first step of the solution to GPT4, the quality of the subsequent steps dramatically improve, and so do the final answer. I guess the hardest part is to understand the problem from scratch and and come up with the right idea.

1 0 1 145 0

View Details

AK @_akhaliq

3 years ago

ARB: Advanced Reasoning Benchmark for Large Language Models paper page: huggingface.co/papers/2307.13… Large Language Models (LLMs) have demonstrated remarkable performance on various quantitative reasoning and knowledge benchmarks. However, many of these benchmarks are losing utility as LLMs get increasingly high scores, despite not yet reaching expert performance in these domains. We introduce ARB, a novel benchmark composed of advanced reasoning problems in multiple fields. ARB presents a more challenging test than prior benchmarks, featuring problems in mathematics, physics, biology, chemistry, and law. As a subset of ARB, we introduce a challenging set of math and physics problems which require advanced symbolic reasoning and domain knowledge. We evaluate recent models such as GPT-4 and Claude on ARB and demonstrate that current models score well below 50% on more demanding tasks. In order to improve both automatic and assisted evaluation capabilities, we introduce a rubric-based evaluation approach, allowing GPT-4 to score its own intermediate reasoning steps. Further, we conduct a human evaluation of the symbolic subset of ARB, finding promising agreement between annotators and GPT-4 rubric evaluation scores.

2 41 174 44K 70

View Details

DuckAI @TheDuckAI

3 years ago

@yacineMTB @stephen_wolfram your software can't defeat me

0 1 9 804 0

View Details

DuckAI @TheDuckAI

3 years ago

Work done by our team at @TheDuckAI and collaborators: @tsawada_ml @dpaleka @dahoas1 @PranavTadepalli @alex_kranias @johnjnay @kshitijkgupta @arankomatsuzaki (4/N)

2 0 10 2K 0

View Details

DuckAI @TheDuckAI

3 years ago

Rubric-based evaluation method for assessing model performance: - Use few-shot prompting to generate rubric from ground truth solution using LLM - Use a LLM to grade a model response based on rubric (3/N)