I would like to be able to log every write/read that my go app issues to the underlying OS, and also (if it's possible) completely replace FS with one that resides only in memory.
Is it possible? How? Maybe there is a ready-to-Go solution?
This is straight from Andrew Gerrand's 10 things you (probably) don't know about Go:
var fs fileSystem = osFS{}
type fileSystem interface {
Open(name string) (file, error)
Stat(name string) (os.FileInfo, error)
}
type file interface {
io.Closer
io.Reader
io.ReaderAt
io.Seeker
Stat() (os.FileInfo, error)
}
// osFS implements fileSystem using the local disk.
type osFS struct{}
func (osFS) Open(name string) (file, error) { return os.Open(name) }
func (osFS) Stat(name string) (os.FileInfo, error) { return os.Stat(name) }
For this to work, you will need to write your code to take a fileSystem argument (maybe embed it in some other type, or let nil denote the default filesystem).
For those looking to solve the problem of mocking out your filesystem during testing, checkout #spf13's Afero library, https://github.com/spf13/afero. It does everything that the accepted answer does, but with better documentation and examples.
You can use the testing/fstest package:
package main
import "testing/fstest"
func main() {
fs := fstest.MapFS{
"hello.txt": {
Data: []byte("hello, world"),
},
}
data, err := fs.ReadFile("hello.txt")
if err != nil {
panic(err)
}
println(string(data) == "hello, world")
}
https://godocs.io/testing/fstest
Just because this question pops up pretty high when googling for this matter:
I don't know about logging reads and writes, but for a filesystem residing only in memory, I've found blang/vfs. I haven't used in production, and it says it's alpha and interfaces are likely to change. Use it at your own risk.
I suppose you could implement it to log reads and writes.
Related
I'm using https://github.com/DATA-DOG/go-sqlmock
and trying to mock a connection to a db.
Now, I need to mock a ping command (for load balancing purposes). However, I can't find any information about how to do that.
For example, I would like to write a test like this
db, mock, _ := sqlmock.New()
// ExpectPing does not exist, but it is there anything similar?
mock.ExpectPing().WillReturnError("mock error")
err := db.Ping()
if err==nil{
t.Fatal("there should have been an error")
}
in addition, I need to inject this mocked object into a function. let's say New(*sql.DB) that outputs a new structure. so it must be compatible with *sql.DB struct. for this reason, sqlmock seems a good choice. However, I've not found the way to mock the ping command.
it is there any way to do this using this library?
if not, is there any other database/sql mock library that could do the trick?
Update: As of Jan 6, 2020, this functionality has been added to go-sqlmock, and is included in the v1.4.0 release.
Original answer:
No, there is "nothing similar." The Ping and PingContext methods depend on the driver implementing the Pinger interface, so you can't mimic it by, for example, expecting a 'SELECT 1' or something.
There is already an issue requesting to add this. It seems to not have gotten much attention. I suspect creating a PR (which is probably about 3 lines of code) would greatly increase the chance of having that feature added.
If your goal is to make a Ping fail, you should be able to mimic that by connecting to an invalid DB endpoint (either with or without sqlmock).
You certainly can mock db itself:
type mockedDB struct {
*sql.DB
}
func (db *mockedDB) Ping() error {
return errors.New("not implemented")
}
func Example_mockedDB_Ping() {
db, _, _ := sqlmock.New()
defer db.Close()
mdb := mockedDB{db}
fmt.Println("mdb.Ping(): ", mdb.Ping())
// Output: mdb.Ping(): not implemented
}
but I don't understand what is the purpose of such experiment.
In the same way you can put mock into new type and define ExpectPing on it.
I needed to mock the Ping() functionality as well. This is how I solved it:
type mockDB struct {
ReturnError error
}
func (db *mockDB) Ping() error {
return db.ReturnError
}
func (db *mockDB) PingContext(ctx context.Context) error {
return db.ReturnError
}
// Pinger to be able to mock & ask just for a ping
type Pinger interface {
PingContext(ctx context.Context) error
Ping() error
}
// DatabasePingCheck returns a Check that validates connectivity to a database/sql.DB using Ping().
func DatabasePingCheck(pinger Pinger, timeout time.Duration) Check {
return func() error {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
if pinger == nil {
return fmt.Errorf("pinger is nil")
}
return pinger.PingContext(ctx)
}
}
I am writing a set of services in Swift 4 that will run on Linux. One of the things I need to do is receive a payload that is digitally signed using the Cryptographic Message Syntax (CMS) format, extract the certificate used to sign it, and then validate the signature. I know that Swift on Linux doesn't contain a Security or CommonCrypto framework for this sort of thing, so I've linked in OpenSSL to try and help with this. I'm about 2 years removed from my C/C++ programming days, so I readily admit I'm in over my head on this portion of the code.
I have 2 simple classes to act as wrappers for OpenSSL BIO and PKCS7 data structures. They look like this:
import Foundation
import OpenSSL
public final class BIOWrapper {
public var bio = BIO_new(BIO_s_mem())
public init(data: Data) {
data.withUnsafeBytes { pointer -> Void in
BIO_write(self.bio, pointer, Int32(data.count))
}
}
public init() {}
deinit {
BIO_free(self.bio)
}
}
public final class PKCS7Wrapper {
public var pkcs7: UnsafeMutablePointer<PKCS7>
public init(pkcs7: UnsafeMutablePointer<PKCS7>) {
self.pkcs7 = pkcs7
}
deinit {
PKCS7_free(self.pkcs7)
}
}
I am able to successfully extract the PKCS#7 container data and validate that the data type code value is NID_pkcs7_signed using this code:
let reqData = Data(bytes: reqBytes)
guard reqData.count > 0 else {
print("Empty request body")
return nil
}
let bioWrapper = BIOWrapper(data: reqData)
guard let container = d2i_PKCS7_bio(bioWrapper.bio, nil) else {
print("No container")
return nil
}
let pkcs7Wrapper = PKCS7Wrapper(pkcs7: container)
let dataTypeCode = OBJ_obj2nid((pkcs7Wrapper.pkcs7.pointee.d.sign).pointee.contents.pointee.type)
print("dataTypeCode : \(dataTypeCode)")
if dataTypeCode == NID_pkcs7_data {
print("GOT DATA!")
} else {
print("Didn't get data")
return nil
}
let pkcs7SignedTypeCode = OBJ_obj2nid(pkcs7Wrapper.pkcs7.pointee.type)
if let signed = pkcs7SignedTypeCode == NID_pkcs7_signed {
print("Signed : \(signed)")
}
However, I've now reached a point where I'm stuck. How can I obtain the X.509 certificate data from the PKCS#7 payload? I can see that the pkcs7Wrapper.pkcs7.pointee.d.sign.pointee.cert data structure should contain the certificate chain data. Its data type is UnsafeMutablePointer<stack_st_x509> and I think I can figure out the code to use OpenSSL's PKCS7_verify method once I get the X.509 certificate data in memory. I just don't know how to do THAT part.
I found this resource that talks about validating receipts on OSX/iOS that touches on a lot of the same issues. They obtain the X.509 certificate from the file system and pass the data into the PKCS7_verify method. I just need to know how to get the certificate data from the PKCS#7 container to pass in.
Can anyone help me with this? I recognize that calling C from Swift is not ideal, but in the absence of a good security/cryptography framework for Swift I'm not aware of any other options.
The core part of the answer is in the code you linked:
let store = X509_STORE_new()
X509_STORE_add_cert(store, appleRootX509)
OpenSSL_add_all_digests()
let result = PKCS7_verify(receiptPKCS7, nil, store, nil, nil, 0)
if result != 1 {
log.atLevelDebug(id: 0, source: "Main", message: "Receipt signature verification failed")
exit(errorCode)
}
What you seem to be missing is the fact that you don't have to extract the X509 certificate from the PKCS7 data yourself. The PKCS7_verify function will do it as part of verification:
An attempt is made to locate all the signer's certificates, first looking in the certs parameter (if it is not NULL) and then looking in any certificates contained in the p7 structure itself. If any signer's certificates cannot be located the operation fails.
Therefore the only certificate you need to load yourself is the root certificate which you have observed they load from the file system in the linked code.
If you still really need a Swift solution to extract the certificate out of the PKCS7 data for some reason, you will have to build an ASN.1 parser for PKCS7. Not sure if this is readily available for Swift, this simple code is what a quick search yielded, and this is a nice description of the PKCS7 data.
Is there a way to to get my autoscaled application's VersionID in my init() function without a Context? The only available option seems to be appengine.VersionID(context.Context). Manually scaled instances have /_ah/start called when they start up (giving access to a Context), but there is nothing like this for autoscaled instances.
I am not caring about the generated ID that appengine.VersionID returns with it, just the app.yaml version.
EDIT: A bit of context: I am wanting to deploy versions in the form x-x-x-dev or x-x-x-live and have my database connection depend on the version suffix. This way, when I look in the GCP console, I can be certain which deployed modules/services are using which database. Of course, I setup my DB connection pool in the init(), which has no access to a Context.
I searched and searched with no answers online anywhere, so here it is.
Simply parse the app.yaml file in your init() function. My example here uses a yaml parsing package, but it can be done more lightweight if you need.
import "github.com/ghodss/yaml"
type AppVersion struct {
Version string `json:"version"`
}
func VersionID() (string, error) {
dat, err := ioutil.ReadFile("app.yaml")
if err != nil {
return "", err
}
a := &AppVersion{}
err = yaml.Unmarshal(dat, a)
if err != nil {
return "", err
}
return a.Version, nil
}
Note that this DOES NOT return the generated ID in the form X.Y that appengine.VersionID() does. Only the X part of the version.
As an aside, in the appengine repo on Github, the actual call to appengine.VersionID requires a Context, but internally calls the internal package with nil. So they basically force you to call it with a Context, but it isn't actually used. It's incredibly infuriating.
EDIT: It should be noted that the new Go SDK in gcloud no longer supports version in the app.yaml, as it is now a required parameter at deploy. However, the "legacy" SDK is still supported and maintained, which I am continuing to use as of today (12/24/2018).
If I set the namespace of my context.Context and call a delay function:
ctx := appengine.NewContext(r)
ctx, err := appengine.Namespace(ctx, "mynamespace")
delayFunc.Call(ctx)
How can I find its name:
var delayFunc = delay.Func("my-func", func(ctx context.Context) {
// How do I extract "mynamespace" from ctx?
})
Is the following acceptable practice?
var delayFunc = delay.Func("my-func", func(ctx context.Context) {
n := datastore.NewKey(ctx, "E", "e", 0, nil).Namespace()
// n == "mynamespace"
})
It works but feels like a hack.
Unfortunately you're out of luck. Appengine does not provide an (exported) API call to access the namespace associated with the context.
The namespace association with the context is handled by the appengine/internal package, but "programs should not use this package directly". The context with namespace is obtained by the internal.NamespacedContext() call, and the namespace "extraction" from context is implemented in internal.NamespaceFromContext(). These are not part of the public API, so you can't (shouldn't) use them.
You basically have 2 options. One is the "hacky" way you presented, which works and you may continue to use it.
Another one is to manually handle it, e.g. by manually putting the namespace into the context with your own key, e.g.:
var namespaceKey = "myNsKey"
ctx = context.WithValue(ctx, namespaceKey, "mynamespace")
And when you need it, you can get it like:
ns := ctx.Value(namespaceKey)
Yes, this has the burden of having to manually update it, and if you'd forgot, you would get an "invalid" or empty namespace. So personally I would go with your "hacky" way for now (until this functionality is added to the public API, if ever).
If you go with the manual way, to get rid of the "risk" factor, you could create a helper function which could take care of this along with the appengine.Namespace() call, so you would not forget about it and it would be safe. It could look like this:
func SetNS(ctx context.Context, ns string) context.Context {
ctx = ctx, err := appengine.Namespace(ctx, ns)
if err != nil {
// handle error
}
ctx = context.WithValue(ctx, namespaceKey, ns)
return ctx
}
And using it:
ctx = SetNS(ctx, "mynamespace")
But it may be a rare case when you need to access namespace from the context, as when you would need it, it might just be enough to pass the context to the proper (Appengine) API call which can extract it from the context.
I am currently trying to test a piece of my code that runs a query on the datastore before putting in a new entity to ensure that duplicates are not created. The code I wrote works fine in the context of the app, but the tests I wrote for that methods are failing. It seems that I cannot access data put into the datastore through queries in the context of the testing package.
One possibility might lie in the output from goapp test which reads: Applying all pending transactions and saving the datastore. This line prints out after both the get and put methods are called (I verified this with log statements).
I tried closing the context and creating a new one for the different operations, but unfortunately that didn't help either. Below is a simple test case that Puts in an object and then runs a query on it. Any help would be appreciated.
type Entity struct {
Value string
}
func TestEntityQuery(t *testing.T) {
c, err := aetest.NewContext(nil)
if err != nil {
t.Fatal(err)
}
defer c.Close()
key := datastore.NewIncompleteKey(c, "Entity", nil)
key, err = datastore.Put(c, key, &Entity{Value: "test"})
if err != nil {
t.Fatal(err)
}
q := datastore.NewQuery("Entity").Filter("Value =", "test")
var entities []Entity
keys, err := q.GetAll(c, &entities)
if err != nil {
t.Fatal(err)
}
if len(keys) == 0 {
t.Error("No keys found in query")
}
if len(entities) == 0 {
t.Error("No entities found in query")
}
}
There is nothing wrong with your test code. The issue lies in the Datastore itself. Most queries in the HR Datastore are not "immediately consistent" but eventually consistent. You can read more about this in the Datastore documentation.
So basically what happens is that you put an entity into the Datastore, and the SDK's Datastore "simulates" the latency that you can observe in production, so if you run a query right after that (which is not an ancestor query), the query result will not include the new entity you just saved.
If you put a few seconds sleep between the datastore.Put() and q.GetAll(), you will see the test passes. Try it. In my test it was enough to sleep just 100ms, and the test always passed. But when writing tests for such cases, use the StronglyConsistentDatastore: true option as can be seen in JonhGB's answer.
You would also see the test pass without sleep if you'd use Ancestor queries because they are strongly consistent.
The way to do this is to force the datastore to be strongly consistent by setting up the context like this:
c, err := aetest.NewContext(&aetest.Options{StronglyConsistentDatastore: true})
if err != nil {
t.Fatal(err)
}
Now the datastore won't need any sleep to work, which is faster, and better practice in general.
Update: This only works with the old aetest package which was imported via appengine/aetest. It does not work with the newer aetest package which is imported with google.golang.org/appengine/aetest. App Engine has changed from using an appengine.Context to using a context.Context, and consequently the way that the test package now works is quite different.
To compliment #JohnGB's answer in the latest version of aetest, there are more steps to get a context with strong consistency. First create an instance, then create a request from that instance, which you can use to produce a context.
inst, err := aetest.NewInstance(
&aetest.Options{StronglyConsistentDatastore: true})
if err != nil {
t.Fatal(err)
}
defer inst.Close()
req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
t.Fatal(err)
}
ctx := appengine.NewContext(req)