Added robots.txt file initial to block ai bots

public/robots.txt

@@ -0,0 +1,23 @@
+# Used for many other (non-commercial) purposes as well
+User-agent: CCBot
+Disallow: /
+
+# For new training only
+User-agent: GPTBot
+Disallow: /
+
+# Not for training, only for user requests 
+User-agent: ChatGPT-User
+Disallow: /
+
+# Marker for disabling Bard and Vertex AI
+User-agent: Google-Extended
+Disallow: /
+
+# Speech synthesis only?
+User-agent: FacebookBot
+Disallow: /
+
+# Multi-purpose, commercial uses; including LLMs
+User-agent: Omgilibot
+Disallow: /