{"id":1059,"date":"2024-03-15T16:28:00","date_gmt":"2024-03-15T16:28:00","guid":{"rendered":"https:\/\/david.bookstaber.com\/?p=1059"},"modified":"2026-09-01T17:19:11","modified_gmt":"2026-09-01T17:19:11","slug":"current-best-solution-for-chat-enabled-services-march-2024","status":"publish","type":"post","link":"https:\/\/david.bookstaber.com\/?p=1059","title":{"rendered":"Current Best Solution for Chat-Enabled Services (March 2024)"},"content":{"rendered":"\n

Published March 2024; competitive landscape has moved significantly since.<\/em><\/p>\n\n\n\n

I\u2019ve been doing some work on new natural-language or \u201cchat-enabled\u201d services for a company in the finance industry. \u00a0In contrast to general chatbots, these services require a Large Language Model (LLM) to be augmented with technical features known as (1) Functions<\/em>, which give the bot access to user-specific data from the company\u2019s API, and (2) Embeddings<\/em>, which enable the bot to incorporate information from proprietary documents. \u00a0The finance industry also demands a high level of security and privacy for any systems that can touch financial or client data.<\/p>\n\n\n\n

Isn\u2019t this just Chat-GPT?<\/em> \u00a0By now everybody is familiar with Chat-GPT: OpenAI\u2019s chatbot interface to the company\u2019s LLMs. \u00a0It\u2019s easy for anyone to license and deploy a chat-enabled service directly from OpenAI. \u00a0But, as I\u2019ll explain shortly, OpenAI doesn\u2019t meet all the requirements here.<\/p>\n\n\n\n

What about open source?<\/em> \u00a0Sometimes it\u2019s better to maintain the control you get with open-source systems than to go with a third-party cloud-based solution. \u00a0And it turns out that the number of open-source LLMs has exploded. \u00a0(For a good time, try sorting through literally hundreds of thousands<\/em> indexed here<\/a>!) \u00a0Oh, but a LLM alone does not complete a natural-language service<\/strong>. \u00a0Creating an effective chatbot interface for a LLM is not trivial: Maintaining a context<\/em> and controlling attention<\/em> are ongoing areas of research and development. \u00a0LLM tuning and prompt engineering can also require extensive and specialized work. \u00a0Techniques for embedding content in a LLM are still rapidly evolving. \u00a0Do you have a team of researchers keeping up with the state of the art in each of these areas? \u00a0A big part of the value in going with a large cloud LLM provider is that these technical details are increasingly being handled under-the-hood in their offerings.<\/p>\n\n\n\n

I looked at technical solutions incorporating both 3rd and 4th generation LLMs. \u00a0What is the difference? \u00a0Gen 3 (example: Chat-GPT 3.5<\/em>) is like having a professional writer who has the reasoning skills of a teenager with ADD. \u00a0Gen 4 is like having a great writer with an advanced degree.<\/p>\n\n\n\n

For now, Gen 4 LLMs are only offered as cloud-based services. \u00a0Most are priced around $20\/MMtoken, except that Google seems intent on undercutting all competition on pricing and is presently under $1\/MMtoken<\/a>.1<\/a><\/sup>\u00a0 Here are the current Gen 4 offerings:<\/p>\n\n\n\n

    \n
  1. Google<\/strong> Gemini Pro<\/a><\/em>: \u00a0Google\u2019s enterprise AI services meet the same levels of enterprise security<\/a> as Google Cloud\u2019s other products. \u00a0I.e., if a company is comfortable storing data in Google Drive then it should be comfortable passing it through Gemini.<\/li>\n\n\n\n
  2. OpenAI<\/strong> GPT-4<\/em>: \u00a0OpenAI\u2019s enterprise services<\/a> are SOC 2 compliant<\/a>. \u00a0However<\/em>, as I discovered when I built a prototype<\/a>, OpenAI\u2019s Functions and Embedding features are still in beta, and the company has been aggravatingly unresponsive to developers wondering when those will be ready for production.<\/li>\n\n\n\n
  3. Anthropic<\/strong> Claude-2.1<\/em>: \u00a0It is unlikely that Anthropic\u2019s current security policies would meet the requirements of the finance industry.<\/li>\n\n\n\n
  4. Other possible options like Falcon-180B<\/em> and Mistral-Large<\/em> are a step down in terms of service levels that can be expected from the likes of OpenAI and Google, and are unlikely to be able to meet the security and privacy requirements of the finance sector.<\/li>\n<\/ol>\n\n\n\n

    All of these services include native support for embeddings and functions.  This is an important consideration because, as mentioned earlier, there is significant technical risk and cost associated with implementing those features using open-source alternatives.<\/p>\n\n\n\n

    Local \/ On-Premises Options<\/strong><\/h3>\n\n\n\n

    Another way to avoid security concerns is to stay off the cloud altogether. \u00a0Presently, no off-cloud LLMs provide Gen 4 performance. \u00a0And getting generalized Gen 3 performance requires serious hardware: Consider two of the most established open-source models that perform at the level of OpenAI\u2019s GPT 3.5<\/em>: Mixtral 8x7B<\/em> and LLaMa-2 70B<\/em>. \u00a0To run at a tolerable speed (about 50 tokens\/second) these require dual RTX 3090 or 4090 GPUs, which presently cost about $2k each, or something like an Apple Silicon M2 Ultra, which is also about $4k.<\/p>\n\n\n\n

    Like all LLMs, open-source models can also be run in the cloud<\/a>. \u00a0As a benchmark for cloud pricing: Mistral\u2019s Gen 3 API is currently $.70\/MMtoken. \u00a0(Note that this is comparable to Google\u2019s price for its Gen 4 API.)<\/p>\n\n\n\n

    Lighter Gen 3 models (e.g., 7B<\/em> versions of LLaMa-2<\/em>, Gemma<\/em>, and Mistral<\/em>) can be tailored to perform well in specific domains, so a chatbot that doesn\u2019t require as much intelligence could probably be developed to run on a typical PC. \u00a0But this would require significant technical risk and time to fine-tune a LLM \u2013 itself a specialized field of expertise.<\/p>\n\n\n\n

    The Winner<\/strong><\/h3>\n\n\n\n

    Right now, Google\u2019s Gemini<\/em> API is the clear solution to support Function- and Embedding-enabled chatbots. \u00a0It offers the latest technology and features, meets the highest standards of security, and also happens to be the cheapest option on the market.<\/p>\n\n\n

    1. Google prices their API by the character, not the token, so I made a rough conversion into token terms. \u00a0One MMtoken is roughly 750,000 words, and that cost is paid for both words sent and received from the LLM. \u00a0Where price is a consideration, all of the Gen 4 APIs make it easy to route usage to their less expensive Gen 3 models if\/when desired. \u21a9\ufe0e<\/a><\/li><\/ol>","protected":false},"excerpt":{"rendered":"

      Published March 2024; competitive landscape has moved significantly since. I\u2019ve been doing some work on new natural-language or \u201cchat-enabled\u201d services…<\/p>\n","protected":false},"author":1,"featured_media":1062,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"[{\"content\":\"Google prices their API by the character, not the token, so I made a rough conversion into token terms. \u00a0One MMtoken is roughly 750,000 words, and that cost is paid for both words sent and received from the LLM. \u00a0Where price is a consideration, all of the Gen 4 APIs make it easy to route usage to their less expensive Gen 3 models if\/when desired.\",\"id\":\"acdd0b0b-b7a7-4278-94ab-40a9564b4ae9\"}]","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":false,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[14,15],"tags":[],"class_list":["post-1059","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","category-finance","post-archive"],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/david.bookstaber.com\/WP\/wp-content\/uploads\/2024\/03\/llm-chatbot-documents-and-datacenter.jpg","_links":{"self":[{"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=\/wp\/v2\/posts\/1059","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1059"}],"version-history":[{"count":3,"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=\/wp\/v2\/posts\/1059\/revisions"}],"predecessor-version":[{"id":1064,"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=\/wp\/v2\/posts\/1059\/revisions\/1064"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=\/wp\/v2\/media\/1062"}],"wp:attachment":[{"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1059"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1059"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/david.bookstaber.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1059"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}