Private GenAI chat with Librechat.ai - Intro

This article is the first part of the series of articles about building your own private ChatGPT like web-chat with Librechat.ai.

If you don’t know what Librechat.ai is, here is a short demo of its capabilities:

The Librechat.ai instance from this video is running on my server and uses LLMs from Amazon Bedrock service. It also uses Google Search API and YouTube Data API to search for information and analyze content of the videos.

Motivation

Why would you need your own ChatGPT? Well, there are several reasons for that:

1. Privacy

By my opinion, this is the most important reason. You don’t want to share your data with third-party services. “What data?”, one may ask. Well, your chat logs, documents, images and any other data you might share with the chat.

When you consume 3rd party services, all your data is stored on their servers, and you have no control over it.

2. Cost

Pay per use only, don’t pay for subscription.

OpenAI charges $20 US per month for its “Plus” plan, which allows you to have access to a model of your choice and have integration with advanced tools. Anthropic “Pro” plan costs $18 US per month.

I and my wife both need advanced plan, it would cost us minimum $36 US per month. And if more family members would join, the cost would grow accordingly. It gets worse if you each needs to use multiple GenAI providers, e.g. both OpenAI and Anthropic.

With Librechat.ai, we can pay per use (for only used tokens) and necessary advanced features we need are provided by the Librechat.ai for free. Running my own solution costs me:

  • About $15 US per month for Amazon Bedrock.
  • $5 US per month for V-Linux server.
  • Google Search API and YouTube Data API are free (I stay within free tier limits).

Total: about $20 US per month, which is almost twice cheaper than two advanced plans.

3. Customization and features

Librechat.ai allows using different models from different providers: OpenAI, Anthropic, Amazon, Azure, Google, etc. You can select providers and models and create specific GenAI agents for most optimal price/performance and price/quality ratio. With Librechat.ai you can have more tools integrated with your chat, than in OpenAI or Anthropic. On top of that you can add any tool you want via OpenAPI. E.g. I use integration with YouTube Data API to analyze content of the videos.

Why not running LLMs on your PC or server?

Some may argue, that the most secure way of GenAI use is to run LLMs on your PC or server and not to pay for 3rd party services. While this is true, it has its own drawbacks:

  • The need for a powerful PC or server to run LLMs. Running LLMs requires powerful GPU (with a lot of Video RAM) and a lot of RAM for CPU. Suitable PC which can handle multiple LLMs requires multiple GPUs, which drives the cost up. Taking into account current GPU prices, we are talking about investments of minimum 3000 USD. In reality even more.
  • Power. Running LLMs on your PC or server requires a lot of power. This means that you will have to pay more for electricity. I live in Germany and electricity is expensive here.

If I would run LLMs on my PC, I would need to build one first, it would cost minimum $3000 US. Estimating its power consumption of about 300W when running LLMs (best case scenario). Running of this PC 8 hours per day would add extra $20 US per month to my electricity bill. If I run it 24/7, my electricity bill would increase by $60 US per month. Comparing this with about $20 US per month for consumption of LLMs from an Amazon Bedrock service, the own PC solution will be more expensive, and I will never return investments into it.

Why I selected Amazon Bedrock?

Here are my reasons:

  • I want to use different LLM models but from one provider, so that I have simpler control over costs and access.
  • My primary model is Anthropic Claude, but I don’t want to overpay for it. Amazon Bedrock prices for Anthropic Claude models is the same as directly on Anthropic itself.
  • Security and privacy.
    • I work with Amazon Web Services (AWS) daily, I know how to securely consume services there.
    • I also trust AWS and I know that content of my messages is not stored in API traces of AWS.

Note: From Data Protection in Amazon Bedrock documentation: “Amazon Bedrock doesn’t store or log your prompts and completions. Amazon Bedrock doesn’t use your prompts and completions to train any AWS models and doesn’t distribute them to third parties.”

Enough talking, let’s build it!

A diagram of the solution is shown below:

Diagram

As you can see from the diagram, the Librechat.ai services are deployed on your own server, but they use LLMs from Amazon Bedrock service and consume Google Search API and YouTube Data API for Google Cloud.

Databases with your chat logs, vector databases (which are used by LLMs to search in your documents) and your uploaded files (documents and images) are stored on your own server. So, Amazon and Google don’t have access to your data.

Prerequisites

  • Your own server with Linux: physical or virtual. Server shall have enough RAM and CPU power to run containers. I personally use V-Linux server with 2 CPUs and 4 GB of RAM. Server shall have:
  • Amazon Web Services Account
  • Google Cloud Account

To be continued…

Before explaining how to install Librechat.ai on your server, we need to obtain necessary access keys and API keys so that we can use LLMs from Amazon Bedrock and Google Search API and YouTube Data API.

Here is my articles plan:

  • Part 2: This part will be about the setup of your AWS account for Amazon Bedrock. It will explain how to request access to LLM models in Amazon Bedrock and how to make the least privileged access to your AWS account for Librechat.ai services.
  • Part 3: This article will be about the setup of a Google Cloud to get your Google Search API and YouTube Data API keys and how to secure access to them.
  • Part 4: Bringing it all together. This part will be about the installation of Librechat.ai services on your server and how to configure them to use LLMs from Amazon Bedrock and Google Search API and YouTube Data API.