75+ Claude AI Model Statistics in Q4 2023

In November 2023, the AI, and at a larger scale, the tech world was feverishly consumed by news of an incipient coup staged by the Open AI (the organization behind ChatGPT) board of directors. Ultimately, that coup failed to result in executive reform. Still, it did provide the perfect opportunity for one of its biggest competitors, Anthropic, to draw attention to its own LLM (large language model). As a result, on Tuesday, Nov 21st, 2023,  Anthropic announced grand updates to its AI model, Claude. Considering the enthusiastic market response, we will look at the many developing trends and statistics behind Claude AI. 

Claude, now in its 2.1 version, is an AI (artificial intelligence) based assistant developed by Anthropic using their research into HHH (Helpful, Honest, Harmless) AI applications. The assistant can be accessed through a chatbot at www.claude.ai or an API (Application Programming Interface) through Anthropic’s developer console. Below, we will explore the interesting data behind Claude AI including development trends, performance metrics, applications and user statistics. 

Development History

Since its inception, Claude has gone through a few upgrades and different iterations: 

• Claude 1.0, anthropic’s first offering was released on March 14, 2023 (source)
• Claude 1.3- April 18, 2023 (source)
• Claude 2.0- July 11, 2023 (source)
• Claude 2.1- November 21, 2023 (source)

Anthropic has also released a streamlined and faster model with limited capabilities, called Claude Instant. This model has seen a similar development cycle as seen below: 

• Claude Instant 1.1- March 14, 2023 (source)
• Claude Instant 1.2- August 9, 2023 (source)

Technical Capabilities 

As mentioned earlier, within the past month, Anthropic has released an upgraded version of their leading AI model, Claude 2.1. Having also made advancements in their Claude Instant model within the last quarter, Anthropic has set its mark as a technical leader in the AI space. Below we can see different stats detailing the capabilities achieved by Claude AI:

• Claude 2 is trained on updated data from 2022 and early 2023 (source)
• Claude 2 can generate documents up to 4000 tokens (roughly 3000 words) (source)
• Claude AI (1.0) reads up to 75,000 words (source)
• Latest model, Claude 2.1, reads up to 150,000 words (source)
• Claude models are trained on over 137 billion text and code parameters (source)
• Claude allows file uploads of PDFs, DOCX, CSV, and TXT formats (source)

Roughly 10% of Anthropic Claude’s dataset comes from non-English content (source)

Performance Metrics 

In developing the different versions of Claude, the Anthropic team extensively tests and measures the performance of their advancing models. Below we can see some recent highlights in Claude AI performance: 

• The overall success of open-sourced LLMs in answering the 858 nephSAP multiple-choice questions correctly was 17.1% – 25.5%. In contrast, Claude 2 answered 54.4% of the questions correctly, whereas GPT-4 achieved a score of 73.3% (source)
• Claude Instant 1.2 scores 58.7% on the Codex evaluation, compared to 52.8% for Claude Instant 1.1. (source)
• It also achieves 86.7% on the GSM8K benchmark, compared to 80.9% for the previous version (source)
• Claude 2 also scored a 71.2% on the Codex HumanEval Python coding test (source)

Claude scored 88.0% on GSM8k grade-school math problems, showcasing its computational ability (source)

Aside from coding performance, Anthropic sees truthfulness, harmlessness, and helpfulness as pillars to Claude’s success. The following benchmarks were recently gathered by the Anthropic team in the wake of Claude 2.1’s release: 

• Claude AI shown to be 80% accurate in human evaluation using multi-turn question answer session (source)
• Claude 2.1 demonstrated a 30% reduction in incorrect answers when compared to claude 2.1 (source)

• Claude 2.1 also has a 3-4x lower rate of mistakenly concluding a document supports a particular claim (source)

Claude 2.1 has also made significant gains in honesty, with a 2x decrease in false statements compared to the previous Claude 2.0 model. (source)

Source: www.anthropic.com

In more traditional benchmarks, the performance of Claude has been monitored while completing arduous standardized exams historically taken by humans. The following statistics measure the progress and improvements made by the different versions of Claude: 

• According to Anthropic, Claude 2 scored 76.5% in the Bar exam's multiple-choice section (GPT-3.5 achieved 50.3%)  (source)

Claude also achieved a score higher than 90% of graduate school applicants in GRE reading and writing exams (source)

Applications & How to Access

Currently, the Claude AI models are available in about 95 countries worldwide (https://www.anthropic.com/claude-ai-locations). If unfortunately, you happen to reside outside those countries, it is still possible to use Claude through one of the methods below: 

• Slack- By using the enterprise version of the slack app, a Claude chatbot can be added to a workspace for free 

• Poe- Through Quora Poe, any user can sign up for a free account to access various LLM AI models including Claude Instant. Claude 2.0 is available to users who sign up for a paid premium account. 
• Vercel AI Playground- Similarly to Quora Poe, Vercel AI Playground allows users to access different LLM AI models. Vercel goes further to provide paid users the opportunity to use Claude and other LLM AI models simultaneously to compare outputs.  

Through the span of 2023, Anthropic has released its various models of Claude to the public. The following below are some quick notes on the current state of the accessibility of the Claude LLM AI models: 

• More than 350,000 people signed up to the waitlist to use Claude 2, the most recent version of the language model. (source)
• Anthropic offers Claude Pro at $20 per month (direct competitor to ChatGPT+) (source)
• Claude Pro allows up to 60,000 queries per month – 5X more than ChatGPT Plus (source)
• Claude Pro users can expect to send at least 100 messages to Claude 2 every eight hours (source)
• Claude does not link conversation logs with usernames, IP addresses, account info or other identifiers (source)
• Conversations remain anonymized (source)
• Complete conversations are temporarily stored to train Claude, but are deleted after 7 days. Only a small sample of conversations may be kept for up to 6 months for R&D (source)
• Claude Instant 1.2 is exclusively available as an API for businesses (source)
• According to our in house research, Claude AI is currently the least blocked LLM by top websites amongst its competitors like Google and OpenAI (source)
• Our study shows that as of December 2023, Claude AI is being blocked by 0.02% of the top 1000 websites blocking AI web crawlers (source)

Pricing

Below is a pricing table comparing the costs of different Claude AI models as of November 2023(source): 

• Claude Instant Avg (Prompt and Completion): $0.0714USD per 7500 words
• Claude 2.0 & 2.1 Avg (Prompt and Completion): $0.16USD per 7500 words

Claude AI Traffic and Reach

As of November, 2023 (latest available data), the Claude AI website has garnered widespread attention, reaching (source): 

• 25.9 million monthly visits 
• 5.2% increase in visits compared to October 2023 
• 42.87% bounce rate 
• 3.31 pages per visit 
• 00:04:38 avg visit duration 
• 2,835th most visited site globally 
• 1,907th most visited site in the United States 
• 51st most visited site in the Programming and Developer Software category 

Claude AI shows great marketing potential as currently, the site reaches roughly 21.68 million users through direct searches, the chart below shows how many users Claude AI reaches through other web traffic sources:

Similarly, this next graph illustrates the traffic driven to Claude AI by different social media channels (source):

• Youtube- 53.34%
• Facebook- 10.9% 
• Twitter/ X- 9.77% 
• Linkedin- 9.68%
• Whatsapp Messenger- 6.1% 
• Other- 10.21%

Claude AI User Demographics

 As of October 2023 (latest available data), the most common Claude AI can be described by the stats below (source): 

Gender: 

• Male- 64.4%
• Female- 36.6%

Age: 

• 18 - 24 =  24.79%
• 25 - 34 = 36.13%
• 35 - 44 = 17.94%
• 45 - 54 = 11.53%
• 55 - 64 = 5.99%
• 65+ = 3.62%

Geography (source): 

• United States- 44.92%
• United Kingdom- 8.45%
• India- 4.21%
• China- 3.09% 
• Taiwan- 3.08% 
• Rest of World- 36.25%

Summary

Time will tell whether Anthropic’s strategic decision to unveil Claude 2.1 during Sam Altman’s skirmish against OpenAI’s board of directors. Resounding market praise and support have made it clear that Anthropic is positioning itself as a frontrunner and key player in the AI field. By focusing on the HHH (honest, harmless, helpful) application of AI, Claude has found a niche in the market which is highlighted by the strength of the statistics listed above. With the continued improvements and advancements showcased by Anthropic and Claude AI, it is evident that AI as a field is at the onset of rapid transformation.