How to update value of two struct efficiently - loops

Question

I have the following code which parses YAML files and needs to match values from one struct external and update the internal struct's type property.
For example, this is the yaml file(translated to bin for simplicity) and content which is parsed correctly
package main
import (
"fmt"
"gopkg.in/yaml.v2"
"log"
)
//internal config model for parsing
type InternalModel struct {
models []Model2 `yaml:"models"`
}
type Model2 struct {
Name string `yaml:"name"`
Type string `yaml:"type"`
Target string `yaml:"target"`
}
var internal_config = []byte(`
models:
- name: myapp
type: app1
target: ./
- name: myapp2
type: app2
target: ./
`)
type ExternalConfig struct {
Landscape Zone `yaml:"Landscape"`
}
type Zone struct {
Zone string `yaml:"zone"`
Models []Model `yaml:"models"`
}
type Model struct {
AppType string `yaml:"app-type"`
ServiceType string `yaml:"service-type"`
}
var external_config = []byte(`
Landscape:
zone: zone1
models:
- app-type: app1
service-type: GCP
- app-type: app2
service-type: AMAZON
zone: zone2
models:
- app-type: app3
service-type: AZURE
- app-type: app4Í
service-type: HEROKU
`)
func main() {
// This is the internal config which needs updated
internalConfiglYaml := InternalModel{}
err := yaml.Unmarshal([]byte(internal_config), &internalConfiglYaml)
if err != nil {
log.Fatalf("error in model internalConfiglYaml: %v", err)
}
//fmt.Printf("%+v\n", internalConfiglYaml)
//--------------------------Second config file-----------------------//
//This is the external config yaml
extConfigYaml := ExternalConfig{}
err = yaml.Unmarshal([]byte(external_config), &extConfigYaml)
if err != nil {
log.Fatalf("error in model extConfigYaml: %v", err)
}
fmt.Printf("%+v\n", extConfigYaml)
landscape := "zone1"
modifiedConfig := ConvertTypes(internalConfiglYaml, extConfigYaml, landscape)
fmt.Printf("%+v\n", modifiedConfig)
}
func ConvertTypes(int_cfg InternalModel, ext_config ExternalConfig, landscape string) (out_cfg InternalModel) {
for _, module := range int_cfg.models {
switch module.Type {
case "app1":
//here I hard-coded the value "GCP" but it should come from the yaml struct after parsing
module.Type = "GCP" // should be something like ext_config.models.service-type when the key in the struct
case "app2":
//here I hard-coded the value "AMAZON" but it should come from the yaml struct after parsing
module.Type = "AMAZON"
}
}
return int_cfg
}
//At the end what I need to do is to get the internal yaml file to be changed to the following struct
//The changes are when the type=app-type I need to modify the type in the internal config, here its GCP and ruby
//internal_config_after_changes := []byte(`
//
//
//models:
// - name: myapp
// type: GCP
// target: ./
//
// - name: myapp2
// type: AMAZON
// target: ./
//
//
//`)
At the end what I need to do is to get the internal yaml file to be changed to the struct above internal_config_after_changes
The changes are when the type=app-type I need to modify the type value in the internal_config, here from app1 to GCP and app2 to amazon
The problem is with the second loop which I should use to iterate on the external_config and the matching values, I'm not sure how to combine them both with efficient way...

Answer 1

Golang FAQ described regarding pointers to maps and slice:
Map and slice values behave like pointers: they are descriptors that
contain pointers to the underlying map or slice data. Copying a map or
slice value doesn't copy the data it points to. Copying an interface
value makes a copy of the thing stored in the interface value. If the
interface value holds a struct, copying the interface value makes a
copy of the struct. If the interface value holds a pointer, copying
the interface value makes a copy of the pointer, but again not the
data it points to.
On iterating through the slice of model inside ConvertType you are actually creating a copy of []Models slice whose value.Type is not changing the value of original struct due that reason.
for _, module := range int_cfg.models{}
Above code snippet is creating a copy of int_cfg.models{}.
Index the slice model to point to the exact underlying array of slice Model to change the value as:
package main
import (
"fmt"
"log"
"strings"
"gopkg.in/yaml.v2"
)
//internal config model for parsing
type InternalModel struct {
Models []Model2 `yaml:"models"`
}
type Model2 struct {
Name string `yaml:"name"`
Type string `yaml:"type"`
Target string `yaml:"target"`
}
var internal_config = []byte(`
models:
- name: myapp
type: app1
target: ./
- name: myapp2
type: app2
target: ./
`)
type ExternalConfig struct {
Landscape []Zone `yaml:"Landscape"`
}
type Zone struct {
Zone string `yaml:"zone"`
Models []Model `yaml:"models"`
}
type Model struct {
AppType string `yaml:"app-type"`
ServiceType string `yaml:"service-type"`
}
var external_config = []byte(`
Landscape:
- zone: zone1
models:
- app-type: app1
service-type: GCP
- app-type: app2
service-type: AMAZON
- zone: zone2
models:
- app-type: app3
service-type: AZURE
- app-type: app4Í
service-type: HEROKU
`)
func main() {
//This is the internal config which needs to be update
internalConfiglYaml := InternalModel{}
err := yaml.Unmarshal(internal_config, &internalConfiglYaml)
if err != nil {
log.Fatalf("error in model internalConfiglYaml: %v", err)
}
fmt.Printf("%+v\n", internalConfiglYaml)
//--------------------------Second config file-----------------------//
//This is the external config yaml
extConfigYaml := ExternalConfig{}
// var response interface{}
err = yaml.Unmarshal(external_config, &extConfigYaml)
if err != nil {
log.Fatalf("error in model extConfigYaml: %v", err)
}
fmt.Printf("%+v\n", extConfigYaml)
landscape := "zone1"
modifiedConfig := ConvertTypes(&internalConfiglYaml, extConfigYaml, landscape)
fmt.Printf("%+v\n", modifiedConfig)
}
// ConvertTypes for changing the intConfig struct types
func ConvertTypes(int_cfg *InternalModel, ext_config ExternalConfig, landscape string) (out_cfg *InternalModel) {
for _, module := range ext_config.Landscape {
if module.Zone == landscape {
for i, value := range module.Models {
switch strings.Compare(value.AppType, int_cfg.Models[i].Type) {
case 0:
//here I hard-coded the value "GCP" but it should come from the yaml struct after parsing
int_cfg.Models[i].Type = value.ServiceType // should be something like ext_config.models.service-type when the key in the struct
default:
}
}
}
}
return int_cfg
}
If you check above code snippet, you will also recognize that I have changed the struct.
type InternalModel struct {
models []Model2 `yaml:"models"`
}
to first letter uppercase to make it exportable as:
type InternalModel struct {
Models []Model2 `yaml:"models"`
}
Because of the struct InternalModel is unexportable field model was unable to parse the provided internal_config yaml which leads to empty []slice data after unmarshal yaml.
One more thing that I have noticed is you are converting bytes into bytes again. There is no need for that.
err := yaml.Unmarshal([]byte(internal_config), &internalConfiglYaml)
So I have changed it to just:
err := yaml.Unmarshal(internal_config, &internalConfiglYaml)
Since internal_config is already declared as byte using []byte in global variable.

Answer 2

If you're looking for how to apply the app-type to service-type mapping from the ExternalConfig, I suggest converting the array into a map and using it to look up the mapping:
func ConvertTypes(intCfg *InternalModel, extCfg ExternalConfig, landscape string) {
// Make an app-type to service-type map.
appToSvc := make(map[string]string, len(extCfg.Landscape.Models))
for _, m := range extCfg.Landscape.Models {
appToSvc[m.AppType] = m.ServiceType
}
// For each InternalModel model, check if there's an app-type to service-type
// mapping. If so, use the service-type instead of the app-type.
for i, model := range intCfg.Models {
if t, ok := appToSvc[model.Name]; ok {
intCfg.Models[i].Type = t
}
}
}
I've also fixed the pass-by-pointer and range loop assignment problems as suggested by #Himanshu and #Zan Lynx.

Answer 3

The first thing to consider is that you are working with two structures that are defined at run-time, thus you will not be able to use a single loop to match them. The next thing to consider is that your InternalConfig is the one least likely to change and should thus be the first structure to parse, where after the ExternalConfig is parsed and matched with it.
From reading your question, comments and the provided answers it also seems that there are a number of assumptions on the board which few have alluded to. I will attempt to address these if possible.
So let's give it a go:
package main
import (
// import standard library packages first (alphabetically)
"fmt"
"log"
// import third party packages next (also alphabetically)
"gopkg.in/yaml.v2"
// finally import your own packages here
)
// InternalConfig
// - Let's call it InternalConfig so as to ensure consistent naming standards
type InternalConfig struct {
Models []Imodel `yaml:"models"`
// Exported to ensure access to external packages (such as yaml)
}
// Imodel - The internal model struct
type Imodel struct {
Name string `yaml:"name"`
Type string `yaml:"type"`
Target string `yaml:"target"`
}
var iCfgb = []byte(`
models:
- name: myapp
type: app1
target: ./
- name: myapp2
type: app2
target: ./
`)
type ExternalConfig struct {
Landscape Zone `yaml:"Landscape"`
}
type Zone struct {
Zone string `yaml:"zone"`
Models []Emodel `yaml:"models"`
}
type Emodel struct {
AppType string `yaml:"app-type"`
ServiceType string `yaml:"service-type"`
}
var eCfgb = []byte(`
Landscape:
zone: zone1
models:
- app-type: app1
service-type: GCP
- app-type: app2
service-type: AMAZON
zone: zone2
models:
- app-type: app3
service-type: AZURE
- app-type: app4Í
service-type: HEROKU
`)
function main() {
iCfgYaml := InternalConfig{}
// don't need []byte(iCfgb) as iCfgb is already a byte slice
// We are also using a shorthand if statement as we will only use err in the
// if scope, this is because our config is passed by reference and not value
if err := yaml.Unmarshal(iCfgb, &iCfgYaml); err != nil {
log.Fatalf("error in model InternalConfig: %v", err)
}
eCfgYaml := ExternalConfig{}
if err := yaml.Unmarshal(eCfgb, &eCfgYaml); err != nil {
log.Fatalf("error in model ExternalConfig: %v", err)
}
// From here there are clearly some assumptions made, here is my take:
// 1. I do not see any purpose in the landscape variable, the value "zone1"
// can be obtained from eCfgYaml.Landscape.Zone;
// 2. From the question it seems you want to update the InternalConfig
// values with values from the ExternalConfig. However, from the code you
// created a modified copy. I will assume this was for debugging purposes
// and skip the creating of another copy;
// I have renamed ConvertTypes to UpdateInternalState to accurately reflect
// the behavior of the function. It now returns an error instead of a copy of
// InternalConfig (any state altering functions should most likely return an
// error). It receives a pointer of InternalConfig and a value copy of
// ExternalConfig. This is because we will be altering the state of
// InternalConfig
if err := UpdateInternalState(&iCfgYaml, eCfgConfig); err != nil {
log.Fatalf("Error updating internal config: %v", err)
}
// Assume the print is for debugging, thus I denote it with DBG. Makes it easy
// to find and remove from production code.
fmt.Printf("DBG: %+v\n", iCfgYaml)
}
func UpdateInternalState(iCfg *InternalConfig, eCfg ExternalConfig) error {
// Firstly, we need to track the internal config models. We will create an
// index lookup map (ilm) for it.
ilm := make(map[string]int)
// Now we iterate through our internal models and update the index:
for i, v := range iCfg.Models {
ilm[v] = i
}
// Now we iterate through the external config and update the internal config
// state accordingly:
for _, em := range eCfg.Landscape.Models {
// We find the index of the specified app type in the index lookup
// We use a shorthand if statement as we will only use i and ok within
// the scope of the if (if you need it outside you can't use shorthand)
// If there is no match nothing happens, but if found it will update to
// the corresponding specified value in the external config.
if i, ok := ilm[em.AppType]; ok {
iCfg.Models[i].Type = em.ServiceType
}
}
// At any point within this function you are able to do more robust error
// checking, such as ensuring the index lookup map must have a corresponding
// entry, and return any errors encountered.
return nil
}

Answer 4

This question is definitely a duplicate but I can't find one to link right now.
Your problem is that the for loop is creating a copy of each struct. Writing into element which is just a copy of xtr[i] never changes the original xtr[i]. What you do to fix it is to use the slice index instead.
Change
for _, element := range xtr {
switch element.vType {
case "app1":
//user1 value is coming from the &y parsed data of the yaml
element.vType = "user1"
case "app2":
//user2 value is coming from the &y parsed data
element.vType = "user2"
}
}
Into
for i := range xtr {
switch xtr[i].vType {
case "app1":
//user1 value is coming from the &y parsed data of the yaml
xtr[i].vType = "user1"
case "app2":
//user2 value is coming from the &y parsed data
xtr[i].vType = "user2"
}
}