Guide

Languages and fallback

A Language is a BCP 47 language tag, kept in its canonical form. English, Dutch and French are built in as Language.English, Language.Dutch and Language.French; any other language is Language(tag), which throws IllegalArgumentException on an invalid tag. Language.parse does the same without throwing.

import dev.taihei.tsuji.*
import dev.taihei.tsuji.Language.*

given Catalogues = Catalogues(CoreMessages.families)

Language("nl-BE").base == Dutch       // true
Language("EN-gb").tag                 // en-GB
Language.parse("not a tag!")          // Left(Not a BCP 47 language tag: 'not a tag!')
Language("de").label.localize(Dutch)  // Duits

label is the language's name as a message, shown in the reader's language: hand-made translations for the built-ins, and the platform's names (the JDK's locale data, or Intl.DisplayNames in the browser) for any other.

A Catalogues registry shows a message in a language L by taking the first template that has its key:

1. L exactly, for example zh-Hant-TW;
2. L without its extensions and private use (-u-…, -x-…), then with its last subtag removed, step by step: zh-Hant, then zh;
3. the fallbacks you configured for L, each walked the same way;
4. the family's source language, which always has the key.

object Greetings extends MessageFamily("greeting"):
  val hello = message("Hello", Dutch -> "Hallo", Language("de") -> "Guten Tag")

given catalogues: Catalogues = Catalogues(
  families = Seq(Greetings),
  fallbacks = Map(Language("lb") -> Seq(Language("de"))),
)

Greetings.hello.localize(Language("nl-BE"))         // Hallo
Greetings.hello.localize(Language("lb-LU"))         // Guten Tag
Greetings.hello.localize(Language("ja"))            // Hello
catalogues.fallbackChain(Language("lb-LU"), English) // List(lb-LU, lb, de, English)

Numbers, dates and plurals are always formatted for L, also when the template comes from a fallback. Showing a message never throws.

Arguments, Text and Displayable

Arguments are typed through the type class MessageArg[A]. It has instances for String, Int, Long, Double, Float, Boolean, LocalDate, LocalDateTime, LocalTime, Language (shown as its name in the reader's language), Message, Text and Displayable (shown in the reader's language), and Option and List of these. A None shows as empty text; a list is joined with , . A type of your own goes through one of them with MessageArg.as:

final case class OrderId(value: String)
given MessageArg[OrderId] = MessageArg.as[OrderId, String](_.value, s => Right(OrderId(s)))

object Orders extends MessageFamily("order"):
  val cancelled = message("Your order was cancelled", Dutch -> "Uw bestelling werd geannuleerd")
  val note      = message[(note: Option[String])]("Note: {$note}")
  val tags      = message[(tags: List[String])]("Tags: {$tags}")
  val failed    = message[(reason: Displayable)]("The order failed: {$reason}")
  val language  = message[(language: Language)]("Shown in {$language}")

given Catalogues = Catalogues(Seq(Orders))

Orders.note(note = None).localize(English)                             // Note:
Orders.tags(tags = List("gift", "express")).localize(English)          // Tags: gift, express
Orders.failed(reason = Orders.cancelled).localize(Dutch)               // The order failed: Uw bestelling werd geannuleerd
Orders.failed(reason = Text("Disk full", English)).localize(Dutch)     // The order failed: Disk full
Orders.language(language = Language("de")).localize(Dutch)             // Shown in Duits

The compiler checks the names and types of the arguments, but not the formatting functions applied to a type of your own: Catalogues.verify checks those in your tests.

Text is text in one language that is not a message of a family: an exception's message, or a text a server already translated. It shows as it is, whatever the reader's language. Text.of(throwable) takes the throwable's message, or its class name when it has none.

Message, Text and the 4.x legacy.Message share the supertype Displayable. Each is shown with localize(language) and a given Catalogues in scope.

MessageFormat 2.0

Templates use the ICU MessageFormat 2.0 syntax, in the dialect this library has always read.

Plural keys are one, other, or an exact number such as =0 or 0. French counts 0 and 1 as one; English and Dutch only 1. A .match on a Language compares its tag. After the declarations, a simple pattern ends at the line break.

object Inbox extends MessageFamily("inbox"):
  val unread = message[(count: Int)](
    ".match {$count :plural} when =0 {No new messages} when one {One new message} " +
      "when * {{$count} new messages}",
    French -> (".match {$count :plural} when one {{$count} nouveau message} " +
      "when * {{$count} nouveaux messages}"),
  )
  val invited = message[(host: String, role: String)](
    ".match {$role :select} when admin {{$host} made you an administrator} " +
      "when * {{$host} invited you}"
  )
  val total = message[(amount: Double)](
    ".local $rounded = {$amount :integer}\nAbout {$rounded} in total"
  )

given Catalogues = Catalogues(Seq(Inbox))

Inbox.unread(count = 0).localize(English)                  // No new messages
Inbox.unread(count = 1).localize(English)                  // One new message
Inbox.unread(count = 5).localize(English)                  // 5 new messages
Inbox.unread(count = 0).localize(French)                   // 0 nouveau message
Inbox.invited(host = "Ada", role = "admin").localize(English)   // Ada made you an administrator
Inbox.invited(host = "Ada", role = "viewer").localize(English)  // Ada invited you
Inbox.total(amount = 41.7).localize(English)               // About 42 in total

MessageFormat2 also formats and checks a template on its own. format never throws, and a variable without a value shows as $name. validate reports syntax errors, unknown functions, a .match without a * variant, and text after the pattern; which variables exist is checked by a message definition, not here.

val tmpl = ".match {$count :plural} when one {You have one item} when * {You have {$count} items}"

MessageFormat2.format(tmpl, Map("count" -> 1), "en")   // You have one item
MessageFormat2.format(tmpl, Map("count" -> 5), "en")   // You have 5 items
MessageFormat2.validate("Hello {$name")                // List(at 6: a placeholder is not closed with })
MessageFormat2.validate(".match {$n :plural} when one {One}")
// List(at 0: .match needs a default variant with * for every selector)

Loading catalogues, and errors

On the JVM

Nothing needs to be configured: files on the classpath are read when a language is first needed. ClasspathCatalogues.read(family, language) reads one file directly: None when there is none, Some(Left(problem)) when it cannot be read, and Some(Right(catalogue)) otherwise.

ClasspathCatalogues.read(Shop, Dutch).map(_.map(_.messages.size))   // Some(Right(2))

A registry remembers the files it found, and of the languages without a file the latest 4096. Narrow a language that comes from a request header to the languages you support before you show messages in it, or each new tag costs a lookup per family.

In Scala.js

The bundle holds the definitions and the source-language templates: one language, not all of them. The application serves the per-language files that the export writes, and loads a language with CatalogueLoader.load(catalogues, language, url). It uses the global fetch, takes a given ExecutionContext, and returns a Future with the problems it found, which never fails. Until a language arrives, its messages show through the fallback chain; onAdded says when to show them again.

import scala.concurrent.ExecutionContext.Implicits.global
import dev.taihei.tsuji.*
import dev.taihei.tsuji.Language.*

val catalogues = Catalogues(Seq(Shop) ++ CoreMessages.families)
given Catalogues = catalogues

def render(language: Language): Unit =
  println(Shop.welcome(name = "Ada").localize(language))

def switchTo(language: Language): Unit =
  render(language)                                  // through the fallback chain until loaded
  CatalogueLoader.load(catalogues, language, s"/i18n/${language.tag}.json").foreach: problems =>
    problems.foreach(p => println(s"could not load ${language.tag}: ${p.message}"))

val stop = catalogues.onAdded(language => render(language))

The library has no dependency on a user interface or an effect library: a Laminar or other front end turns the onAdded callback into its own signal. On a language switch the application fetches the next file; it may prefetch one, the library does not.

Errors

Showing a message never throws: a missing or broken translation falls back, and the source language always has the key. What went wrong is a CatalogueProblem, with a message, an optional language and an optional key. Input text in a problem is escaped and cut short, so it is safe to log. You find them in what Catalogue.parse, add, CatalogueLoader.load and verify return, and in catalogues.problems (the first 1000).

Catalogue.parse("""{"format":2,"language":"fr","messages":{}}""")   // Left(format 2 is not format 1)
Catalogue.parse("not json")                                        // Left(not JSON: (expected 'null'))

Exporting catalogues

taihei-tsuji-tools has one program, dev.taihei.tsuji.tools.ExportCatalogues. It finds the families on the classpath, those of libraries such as CoreMessages included, and writes one merged file per language, <out>/<tag>.json. The file of the source language is the one to hand to translators; the others hold the catalogue files on the classpath over the inline translations, ready for a browser application to load.

ExportCatalogues --out <dir> [--scan <package>]... [--family <object>]...

--scan finds every family object in a package; --family names one. Templates that fail their checks are left out and printed, and make the exit status 1; a usage error exits with 2. It runs on the JVM, in a module whose classpath holds the families: for a cross-built application, the JVM side of its shared code. A family defined only in Scala.js code cannot be exported.

// Mill: the tools on the run classpath of the module that holds the families
def runMvnDeps = Seq(mvn"dev.taihei::taihei-tsuji-tools:5.0.0")
// ./mill app.runMain dev.taihei.tsuji.tools.ExportCatalogues --out out/tsuji --scan com.example --scan dev.taihei.tsuji

// sbt: run it forked, so that its classpath is the application's
libraryDependencies += "dev.taihei" %% "taihei-tsuji-tools" % "5.0.0"
Compile / run / fork := true
// sbt "runMain dev.taihei.tsuji.tools.ExportCatalogues --out target/tsuji --scan com.example --scan dev.taihei.tsuji"

Run against the Shop family of Getting started, in a package com.example.shop and with an inline French translation of welcome:

2 messages in de: out/tsuji/de.json
2 messages in en: out/tsuji/en.json
1 messages in fr: out/tsuji/fr.json
2 messages in nl: out/tsuji/nl.json

{
  "format" : 1,
  "language" : "fr",
  "messages" : {
    "shop.welcome" : "Bon retour, {$name}"
  }
}

Add --scan dev.taihei.tsuji to export the Dutch and French catalogues of CoreMessages with your own, which a browser application needs to show them in those languages.

Ready-made messages

CoreMessages holds families of common messages, with English as the source language and Dutch and French as catalogue files in the jar: General, Authentication, Authorization, Validation, Exceptions, and Languages, which Language.label uses. CoreMessages.families lists them.

given Catalogues = Catalogues(CoreMessages.families)

CoreMessages.Validation.stringMinLength(min = 8).localize(Language.Dutch)
// lengte moet minimaal 8 zijn
CoreMessages.Validation.formatErrors(Map(
  "email" -> List(CoreMessages.Validation.stringValidEmail),
  "name"  -> List(CoreMessages.Validation.stringShouldNotBeBlank),
)).localize(Language.French)
// Échec de validation: email (Doit être une adresse e-mail valide), name (ne doit pas être vide)

In Scala.js the Dutch and French files reach the browser through the export, like your own; until they are loaded, these messages show in English. The full list of members is in the CoreMessages reference.

Limits on untrusted input

Catalogue files and JSON messages may come from outside your code: a translator, a tenant, a client. Tsuji bounds what it reads and what it produces, and refuses or falls back past a bound instead of running out of memory or time:

• A catalogue is at most 4M characters (a classpath file at most 16 MB, UTF-8 only), 100000 messages and 16 levels of JSON nesting; a template in it at most 64K characters. Entries past the template bound are left out and listed as problems.
• A catalogue in which more than 16 keys share one hash code is refused. Real keys almost never share one, and a map of keys that do costs time out of proportion to its size. The same bound applies to the names in one object of the kept arguments of a message with an unknown key.
• Messages nest at most 8 deep, a list argument holds at most 10000 elements, and the kept arguments of a message with an unknown key are at most 16 levels and 100000 values.
• A 4.x message brings at most 1000 templates and 1000 arguments, none of them longer than 64K characters.
• One template formats to at most a million characters, and one message with its nested messages, list elements and fallbacks to at most four million; past that it shows its own template.
• On the JVM a registry remembers the latest 4096 languages without a file; narrow a language from a request to the ones you support.

val tooLong = " " * (4 * 1024 * 1024 + 1)
// colliding: a catalogue in German whose 17 keys share one hash code

Catalogue.parse(colliding)   // Left(de: more than 16 keys of a catalogue share a hash code)
Catalogue.parse(tooLong)     // Left(a catalogue is at most 4194304 characters)

taihei-i18n 4.0.1 and earlier had no such bounds for Message JSON; see advisory GHSA-6h29-v54g-8337. Report a security issue to security@taihei.dev, as the security policy describes.