How to Migrate from Amazon MSF

Chances are you’ll have credentials you wouldn’t want to hard-code into your Flink application. Amazon MSF has its own way of handling credentials for Flink applications , including using the AWS Secrets Manager.

When running your Flink application on Decodable, you use Decodable’s support for secrets . Once created, these secrets can be accessed directly on the filesystem from your code, or using the Decodable SDK .

There are parameters that your application will use at runtime that may be specific to an environment. For example, in development, the host or schema of a database being accessed may differ from that used when the application runs in production. Instead of hardcoding this into your Flink application and having to recompile it for each environment (which adds additional complexity and opportunity for mistakes to arise), you would define these as runtime variables.

In Amazon MSF, runtime configuration is available through runtime properties . The configuration is then accessed in your application as detailed in the docs . In Decodable, you pass configuration values to your Flink application using configuration files . After creating one or more configuration files and adding them to your pipeline, you can access these properties from within your Flink application by reading them from the filesystem under /opt/pipeline-config. Since it’s just a file, you can use any configuration library or custom code.

With Decodable, along with configuration files, you can also pass arguments to your Flink job.

You might use job arguments to specify an environment. For example, you can pass an argument to your application to ignore test data. Another use might be to give your application the ability to reset and ignore existing data.

This is done with the Job arguments setting for your custom pipeline. These can be read by your application using Flink’s ParameterTool class ._ _Using job arguments also has the nice advantage that your job behaves the same during local testing as when you run it in the managed service environment—without requiring any special handling as is required on Amazon MSF .

While logs are typically consumed via Amazon CloudWatch when running Flink jobs on Amazon MSF, Decodable offers access to logs through a CLI . Access to the Flink web interface is also available while running jobs on Decodable, in case you have operational workflows relying on this access.

Metrics are written to a stream in Decodable, which you can preview from the CLI or web interface. _You can also use the stream as an input to one of the many pre-built connectors that Decodable offers and then route the metric data to an external system such as Datadog .