Using Temporal Cloud With On-Prem Data

Using cloud services is standard practice for most backend application architectures. When using cloud services, it is important to understand and control what data is leaving your network and being sent to the cloud. Temporal Cloud has great options available to ensure that data sent to and from the cloud is securely encrypted. This post will showcase how Temporal Cloud might interact with your infrastructure by default and how you can customize Temporal to prevent any user or business-related data from being sent to the cloud.

Understanding Temporal Cloud

Cloud services can be difficult to understand without proper visualization. We created an image to help explain how Temporal Cloud manages data.

While Temporal Workers are executing a Workflow, information will be sent into the Temporal Cloud and returned to your server or service upon completion. Now that you can visualize the path of the data, let’s tackle the issue of security and encryption when dealing with information sent to and from the Temporal Cloud.

Defining Your Own DataConverter Option for Your Temporal Client

Temporal provides a converter library that you can import in Go to create your own DataConverter to pass as an option to your Temporal Workers and Client. This customized data converter extends a PayloadCodec interface with two methods: Encode and Decode. These methods will be called to encode and decode the data that passes through the Temporal Workers, as well as the Temporal Client you create. Here is a high-level (and simplified) look at what it will look like:

Worker and Temporal Client Integration

When instantiating the Temporal Client, there is an option to create your own DataConverter. You will be creating a method called NewEncryptionDataConverter, where you will define your own Codec to be used in the data conversion process.

NewEncryptionDataConverter Method

This method will return a *DataConverter struct, which is from the go.temporal.io/sdk/converter library. Pass in your own Codec struct, which you define to extend the PayloadCodec type. Your Codec struct has two methods: Encode and Decode, which we will review later in this post.

See below what this method can look like:

func NewEncryptionDataConverter(dataConverter converter.DataConverter, options DataConverterOptions) *DataConverter {
	codecs := []converter.PayloadCodec{
		&Codec{KeyID: options.KeyID},
	}

	return &DataConverter{
		parent:        dataConverter,
		DataConverter: converter.NewCodecDataConverter(dataConverter, codecs...),
		options:       options,
	}
}

The key in the codec will be used to encrypt and decrypt the data input to the DataConverter. You may need to change how the key is passed within the DataConverter according to your security regulations or preferences. Then, finally, return the DataConverter struct that the Temporal Client needs.

The codec struct looks like this:

// Codec implements PayloadCodec using AES Crypt.
type Codec struct {
	KeyID string
}

Encode Method

// Encode implements converter.PayloadCodec.Encode.
func (e *Codec) Encode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) {
	result := make([]*commonpb.Payload, len(payloads))
	for i, p := range payloads {
		origBytes, err := p.Marshal()
		if err != nil {
			return payloads, err
		}

		key := e.getKey(e.KeyID)

		b, err := encrypt(origBytes, key)
		if err != nil {
			return payloads, err
		}

		result[i] = &commonpb.Payload{
			Metadata: map[string][]byte{
				converter.MetadataEncoding: []byte(MetadataEncodingEncrypted),
				MetadataEncryptionKeyID:    []byte(e.KeyID),
			},
			Data: b,
		}
	}

	return result, nil
}

The code above takes a protobuf payload that Temporal defined (commonpb.Payload). You retrieve the key needed to encrypt the payload, then the code returns a struct following the commonpb.Payload structure.

Decode Method

// Decode implements converter.PayloadCodec.Decode.
func (e *Codec) Decode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) {
	result := make([]*commonpb.Payload, len(payloads))
	for i, p := range payloads {
		// Only if it's encrypted
		if string(p.Metadata[converter.MetadataEncoding]) != MetadataEncodingEncrypted {
			result[i] = p
			continue
		}

		keyID, ok := p.Metadata[MetadataEncryptionKeyID]
		if !ok {
			return payloads, fmt.Errorf("no encryption key id")
		}

		key := e.getKey(string(keyID))

		b, err := decrypt(p.Data, key)
		if err != nil {
			return payloads, err
		}

		result[i] = &commonpb.Payload{}
		err = result[i].Unmarshal(b)
		if err != nil {
			return payloads, err
		}
	}

	return result, nil
}

The decode method takes a payload matching the specific encoding that was provided in the Encode method, using the key to decrypt the data and return it in the same protobuf style.

Adding a DataStore to Your DataConverter

Let’s say you wanted to integrate our own hosted MongoDB (this can be a SQL DB, Redis, S3, or any datastore you choose). Your DataConverter will then store the information during the encoding process and return an encrypted ID to reference the information inside your datastore. At a high level, this is how it will look:

The Encode and Decode methods must be updated to include the MongoDB dependency.

Updating the Codec Struct

Start by updating the Codec struct within the library containing your NewDataConverter method. This way, you can access your MongoDB datastore.

// Codec implements PayloadCodec using AES Crypt.
type Codec struct {
	KeyID string
	Db    *mongo.Controller
}

Changes to NewEncryptionDataConverter

Next, change the NewEncryptionDataConverter method. You’ll only have to change two lines in the method: creating a MongoDB instance and passing it in as your Dbfield in the Codec struct.

func NewEncryptionDataConverter(dataConverter converter.DataConverter, options DataConverterOptions) *DataConverter {
	mongoController := mongo.NewMongoController()
	codecs := []converter.PayloadCodec{
		&Codec{KeyID: options.KeyID, Db: mongoController},
	}
	... rest of your code
}

In this example, you have a basic MongoDB controller that you define, the code for which can be seen at the end of the blog post.

Changes to the Encode Method

Here, you’ll create your own UUID, insert the record in your own MongoDB datastore using the UUID as the primary key, and send the encrypted UUID to Temporal Cloud.

func (e *Codec) Encode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) {
	result := make([]*commonpb.Payload, len(payloads))
	for i, p := range payloads {
		//create uniqueID to return
		uuidToCodex := uuid.New()
		dataToInsert := make(map[string]interface{}, 0)
		err := json.Unmarshal(p.Data, &dataToInsert)
		if err != nil {
			return payloads, err
		}

		key := e.getKey(e.KeyID)

		// insert record into db
		dataToInsert["_id"] = uuidToCodex.String()

		e.Db.InsertRecord("codex-data", dataToInsert)

		//return the encrypted uuid back
		b, err := encrypt([]byte(uuidToCodex.String()), key)

		if err != nil {
			return payloads, err
		}

		result[i] = &commonpb.Payload{
			Metadata: map[string][]byte{
				converter.MetadataEncoding: []byte(MetadataEncodingEncrypted),
				MetadataEncryptionKeyID:    []byte(e.KeyID),
			},
			Data: b,
		}
	}

	return result, nil
}

Changes to the Decode Method

Once you decrypt the payload, use the UUID to retrieve the record from the collection in the datastore and then return the data as a commonpb.Payload type. It’s crucial to encode your result in "json/plain"; otherwise, the Workflow will have encoding errors. What you’ll see returned in your infrastructure is the exact data that we inputted for the workflow.

func (e *Codec) Decode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) {
	result := make([]*commonpb.Payload, len(payloads))
	for i, p := range payloads {
		...code as before
		b, err := decrypt(p.Data, key)
		if err != nil {
			return payloads, err
		}
		result[i] = &commonpb.Payload{}

		// retrieve record by converting []byte into the decrypted UUID
		storedObj, err := e.Db.RetrieveRecord("codex-data", string(b))
		if err != nil {
			return payloads, err
		}

		payload, err := json.Marshal(&storedObj)
		if err != nil {
			return payloads, err
		}
		
    // Metadata, on return the MetadataEncoding is "json/plain"
		result[i] = &commonpb.Payload{
			Metadata: map[string][]byte{
				converter.MetadataEncoding: []byte(converter.MetadataEncodingJSON),
			},
			Data: payload,
		}
	}

	return result, nil
}

Conclusion

This post covered the process of setting up your own DataConverter method in your infrastructure and employing your own encode/decode methods to match your business/regulatory requirements. From there, we reviewed how to reduce the information you send to Temporal Cloud by using your own datastore, sending only an encrypted reference to that data. With this method, you can use Temporal Cloud even if you have regulatory or security requirements that prevent you from sending information outside of your infrastructure. You can find a link to the GitHub repo with the sample code here: Using Temporal Cloud with On-Prem Data Code Samples.

Need more help with Temporal Cloud?

Bitovi is an official Temporal.io partner, and we offer free Temporal audits to new clients. Schedule a consultation for expert help with your Temporal implementation.