Skip to main content

Converters and encryption - Ruby SDK

Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures.

A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these.

The server itself never adds encryption over Payloads. Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. When working with sensitive data, you should always implement Payload encryption.

Custom Payload Codec​

Custom Data Converters can change the default Temporal Data Conversion behavior by adding hooks, sending payloads to external storage, or performing different encoding steps. If you only need to change the encoding performed on your payloads -- by adding compression or encryption -- you can override the default Data Converter to use a new PayloadCodec.

The Payload Codec needs to extend Temporalio::Converters::PayloadCodec and implement encode and decode methods. These should convert the given payloads as needed into new payloads, using the "encoding" metadata field. Do not mutate the existing payloads. Here is an example of an encryption codec that just uses base64 in each direction:

class Base64Codec < Temporalio::Converters::PayloadCodec
  def encode(payloads)
    payloads.map do |p|
      Temporalio::Api::Common::V1::Payload.new(
        # Set our specific encoding. We may also want to add a key ID in here for use by
        # the decode side
        metadata: { 'encoding' => 'binary/my-payload-encoding' },
        data: Base64.strict_encode64(p.to_proto)
      )
    end
  end

  def decode(payloads)
    payloads.map do |p|
      # Ignore if it doesn't have our expected encoding
      next p unless p.metadata['encoding'] == 'binary/my-payload-encoding'

      Temporalio::Api::Common::V1::Payload.decode(
        Base64.strict_decode64(p.data)
      )
    end
  end
end

Set Data Converter to use custom Payload Codec

When creating a client, the default DataConverter can be updated with the payload codec like so:

my_client = Temporalio::Client.connect(
  'localhost:7233',
  'my-namespace',
  data_converter: Temporalio::Converters::DataConverter.new(payload_codec: Base64Codec.new)
)
  • Data encoding is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted.
  • Data decoding may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the temporal workflow show command or when browsing the Web UI to view output.

Using a Codec Server​

A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. Refer to the Codec Server documentation for information on how to design and deploy a Codec Server.

Payload conversion​

Temporal SDKs provide a default Payload Converter that can be customized to convert a custom data type to Payload and back.

Conversion sequence​

The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. You can set multiple encoding Payload Converters to run your conversions. When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion.

Payload Converters can be customized independently of a Payload Codec. Temporal's Converter architecture looks like this:

Temporal converter architecture

Temporal converter architecture

Supported Data Types​

Data converters are used to convert raw Temporal payloads to/from actual Ruby types. A custom data converter can be set via the data_converter keyword argument when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters. Payload converters convert Ruby values to/from serialized bytes. Payload codecs convert bytes to bytes (e.g. for compression or encryption). Failure converters convert exceptions to/from serialized failures.

Data converters are in the Temporalio::Converters module. The default data converter uses a default payload converter, which supports the following types:

  • nil
  • "bytes" (i.e. String with Encoding::ASCII_8BIT encoding)
  • Google::Protobuf::MessageExts instances
  • JSON module for everything else

This means that normal Ruby objects will use JSON.generate when serializing and JSON.parse when deserializing (with create_additions: true set by default). So a Ruby object will often appear as a hash when deserialized. Also, hashes that are passed in with symbol keys end up with string keys when deserialized. While "JSON Additions" are supported, it is not cross-SDK-language compatible since this is a Ruby-specific construct.

The default payload converter is a collection of "encoding payload converters". On serialize, each encoding converter will be tried in order until one accepts (default falls through to the JSON one). The encoding converter sets an encoding metadata value which is used to know which converter to use on deserialize. Custom encoding converters can be created, or even the entire payload converter can be replaced with a different implementation.

NOTE: For ActiveRecord, or other general/ORM models that are used for a different purpose, it is not recommended to try to reuse them as Temporal models. Eventually model purposes diverge and models for a Temporal workflows/activities should be specific to their use for clarity and compatibility reasons. Also many Ruby ORMs do many lazy things and therefore provide unclear serialization semantics. Instead, consider having models specific for workflows/activities and translate to/from existing models as needed. See the next section on how to do this with ActiveModel objects.

ActiveModel​

By default, ActiveModel objects do not natively support the JSON module. A mixin can be created to add this support for ActiveModel, for example:

module ActiveModelJSONSupport
  extend ActiveSupport::Concern
  include ActiveModel::Serializers::JSON

  included do
    def as_json(*)
      super.merge(::JSON.create_id => self.class.name)
    end

    def to_json(*args)
      as_json.to_json(*args)
    end

    def self.json_create(object)
      object = object.dup
      object.delete(::JSON.create_id)
      new(**object.symbolize_keys)
    end
  end
end

Now if include ActiveModelJSONSupport is present on any ActiveModel class, on serialization to_json will be used which will use as_json which calls the super as_json but also includes the fully qualified class name as the JSON create_id key. On deserialization, Ruby JSON then uses this key to know what class to call json_create on.

Use a custom Payload Codec​

A custom Payload Codec can be used to transform an array of Payloads (for example, for encryption or compression) in the Client and the Worker Process.