Connecting KSQL to a Secured Schema Registry

Confluent Cloud now includes a secured Schema Registry, which you can use from external applications, including KSQL.

To configure KSQL for it you need to set:

ksql.schema.registry.url=https://<Schema Registry endpoint>
ksql.schema.registry.basic.auth.credentials.source=USER_INFO
ksql.schema.registry.basic.auth.user.info=<Schema Registry API Key>:<Schema Registry API Secret>

If you’re using Docker Compose then the equivilent of the above is:

KSQL_KSQL_SCHEMA_REGISTRY_URL: "https://<Schema Registry endpoint>"
KSQL_KSQL_SCHEMA_REGISTRY_BASIC_AUTH_CREDENTIALS_SOURCE: "USER_INFO"
KSQL_KSQL_SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO: "<Schema Registry API Key>:<Schema Registry API Secret>"

If you don’t set this correctly then you might get this kind of semi-cryptic error from KSQL when you try to read or write Avro data:

Could not connect to Schema Registry service: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column: 2]; error code: 50005

Schema registry fetch for topic pksqlc-l5z62WIKIPEDIA_FLATTENED request failed.

Caused by: Unexpected character ('<' (code 60)): expected a valid value (number,
        String, array, object, 'true', 'false' or 'null')
 at [Source:
        (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column:
        2]; error code: 50005